Add nodes ip resolve to HA's api-server's /etc/hosts

Hi, After create the HA，I notice that when I use kubectl port-forward or exec to some container, if the container locates on other nodes rather than on api-server node, dns lookup fails. The message shows that:

kubectl port-forward -n kube-system service/kubernetes-dashboard 10443:443
error: error upgrading connection: error dialing backend: dial tcp: lookup machine5 on 127.0.0.53:53: server misbehaving

After some digging, I found that after microk8s join, the node only add mater’s ip and itself’s ip to the /etc/hosts, so lookup the hostname of other node would failed due to the config of the lan router.

So I add all nodes to the /etc/file of the node which api-server is on.

So maybe add all nodes hostname automaticly to /etc/hosts would be nice.

Thanks in advance

[PAStheLoD]

The problem is that CoreDNS does not resolve the hostname of the nodes (and even though the feature was requested it was marked as WONTFIX basically), and thus there really isn’t any service that knows it besides the k8s apiserver.

Editing the hosts file is not ideal, but at least can be done with this on each node:

microk8s kubectl get nodes -o wide | awk '{ print $6 " " $1 }' | tail -n +2 | grep -v $(hostname)  >> /etc/hosts

If your nodes going to change frequently, then probably you need to put something like this into a CRON script (or systemd timer unit). It’s the same as the previous one-liner with before-after guard lines to be able to replace multiple lines.

perl -i -p0e 's/## microk8s nodes begin.*## microk8s nodes end//s' /etc/hosts
echo '## microk8s nodes begin' >> /etc/hosts
microk8s kubectl get nodes -o wide | awk '{ print $6 " " $1 }' | tail -n +2 | grep -v $(hostname)  >> /etc/hosts
echo '## microk8s nodes end' >> /etc/hosts

It’s also possible to somehow put all this into the Corefile of CoreDNS (which is basically in the coredns configmap in the kube-system NS), and use the in-cluster DNS service from the hosts, but it’s left as exercise for the reader, because it seems even more fragile than manipulating the hosts file.

[tobiasmuehl]

To give some weight to this issue, it is one of only 2 reasons why I migrated everything from microk8s to k3s