Unable to get pod logs

I have just installed a fresh snap of Microk8s on the 1.14/stable channel. I am completely unable to get logs.

On the host machine, I am trying to run this command:

microk8s.kubectl logs -n kube-system pod/hostpath-provisioner-6d744c4f7c-m9swk

However the same result is true for any pod I wish to inspect. The result is that I immediately receive this error message:

F0507 11:53:00.594460       1 hostpath-provisioner.go:162] Error getting server version: Get https://10.152.183.1:443/version: dial tcp 10.152.183.1:443: i/o timeout

This error is very strange to me. The IP of my machine is NOT 10.152.183.1. It’s actually 192.168.1.217. I don’t use the 10.x.x.x scheme on my network at all, so I have no idea where it’s getting that number from. Moreover, I’m doing this from the host machine itself, so I would have expected it to try 127.0.0.1. Why don’t logs work?

[soapergem]

I believe @haphan was correct and that it was indeed firewall settings. Applying the suggested ufw changes seems to have fixed the problem. I also had to run sudo iptables -P FORWARD ACCEPT on the host machine. I’ll close this issue now as I’ve got it working.

[haphan]

@ktsakalozos I was facing the same issue recently; pod/hostpath-provisioner is in CrashLoopBackOff state and getting logs throwing a similar error.

After pulling my hair off, I figured out it has something to do with the local firewall settings. Enabling ip forwarding fixed the issue in my case (I am using firewalld instead of ufw)

sudo firewall-cmd --zone=trusted --add-masquerade --permanent
sudo firewall-cmd --zone=trusted --add-interface=cbr0 --permanent
sudo firewall-cmd --reload

Alternatively, for those who use ufw, this shall work as well as already mentioned in the official doc

sudo ufw allow in on cbr0 && sudo ufw allow out on cbr0
sudo ufw default allow routed