Unable to connect to the server: unexpected EOF

I have used microk8s for a while on a single node to host some production workloads. Today I needed to check logs in one of the pods and discovered that while most things seem to be working fine, the logs and exec actions fail. Other actions like create and apply work fine.

[jonathan@zeus ~]$ microk8s kubectl -n camerahub-preprod get pods
NAME                               READY   STATUS      RESTARTS   AGE
postgres-backup-1606100400-b9c7n   0/1     Completed   0          18h
postgres-backup-1606014000-dktmd   0/1     Completed   0          42h
postgres-backup-1605927600-5dp8f   0/1     Completed   0          2d18h
redis-cluster-0                    1/1     Running     4          4d10h
postgres-replica-0                 1/1     Running     4          4d10h
redis-cluster-1                    1/1     Running     4          4d10h
postgres-0                         1/1     Running     4          4d10h
camerahub-b5dbb7676-698x6          0/1     Running     7          4d
camerahub-6cfc8d9cd8-bbplj         0/1     Running     9          52m

[jonathan@zeus ~]$ microk8s kubectl -n camerahub-preprod logs camerahub-6cfc8d9cd8-bbplj
Unable to connect to the server: unexpected EOF

[jonathan@zeus ~]$ microk8s kubectl -n camerahub-preprod exec -it camerahub-6cfc8d9cd8-bbplj -- sh
W1123 21:29:56.949157  552486 http.go:424] Error reading backend response: unexpected EOF
                                                                                         error: error sending request: Post "https://127.0.0.1:16443/api/v1/namespaces/camerahub-preprod/pods/camerahub-6cfc8d9cd8-bbplj/exec?command=sh&container=camerahub&stdin=true&stdout=true&tty=true": unexpected EOF

I haven’t found much on Google about this error. I have tried stopping and starting microk8s and rebooting the server, which did not help. I have not yet attempted to runmicrok8s reset as I am cautious about some of my production PVs - I assume they can be remounted but I’m not 100% sure how that works yet.

inspection-report-20201123_213231.tar.gz

[djjudas21]

Just to update on this - I wiped and reinstalled zeus, set up dnsmasq again in the normal way with a static IP and providing DHCP+DNS to the rest of the LAN, and then installed MicroK8s but did not yet join it to a cluster. Straight away it was showing this error with kubectl logs. I joined it to my cluster, and of course the error persisted.

So I set up dnsmasq on a Raspberry Pi and then rebuilt zeus without dnsmasq, using DHCP to get an IP. This time MicroK8s worked in standalone mode, and continued to work in clustered mode. So it looks like the presence of dnsmasq on the system somehow interferes with libnss_dns.so.2.

I’m happy to accept that this problem is mostly due to my environment - however if there is something wrong with DNS resolution then I would expect the error message to say it was unable to look something up, rather than crashing with a library error. So maybe there’s an underlying bug that is being exposed by my environment? Or maybe an opportunity for a documentation improvement to warn about interactions with dnsmasq?

I am no longer seeing this problem now I’ve moved dnsmasq away from MicroK8s but I am still happy to help track this down if it is useful to the MicroK8s project. Thanks for the help so far 👍

[mattiasarro]

For what it’s worth, in my case this was caused by proxy settings. kubectl logs fetches logs from kubelet at hostname:10250, and my proxy configuration was such that hostname:10250 went to the proxy rather than localhost:10250. Adding hostname to no_proxy env var (in /etc/environment) and restarting microk8s did the trick.