TaintManagerEviction - Pod gets new IP 1 to a few times a day to a few days appart - preceeded by Warning Unhealthy pod/calico-kube-controllers

I’ve asked on stack overflow and no one can help. So treat this as a request to enhance the document. 25% chance there is a bug in microk8s.

Microk8s relaunches a pod and gives it a new IP address every 8-24hrs. The pod does not receive or send traffic at the time of being recreated or generate any logs before microk8s relaunches the pod. microk8s kubectl logs <podname> --follow does not show any logs post starting the experiment, it just stops in the middle of the night.

microk8s kubectl logs <podname> -p only shows normal operation before is relaunched by microk8s.

microk8s kubectl get pods reports an AGE for the pod as if it was not restarted.

The same docker image runs months on docker without any crashes.

I’ve searched through the inspect logs and do not understand these logs or where to focus. So I’m effectively blind. This blind man speculates that IP leases set up by microk8s expire and microk8s relaunched the pod so they use the new IP.

Just in case it’s related to the Ubuntu 18 PC, I reformated the drive, installed ubuntu 20, microk8s, and the problem persists.

Edit: Jan 5, 2021: microk8s kubectl get events mostly returns “no resources found in default namespace” but did report TaintManagerEviction, see jan 5 updates below

inspection-report-20210104_122244.tar.gz

BTW, microk8s is way cool and easy to setup.

[ktsakalozos]

I do not think the snap.microk8s.daemon-apiserver-kicker is causing this. The snap.microk8s.daemon-apiserver-kicker [1] is keeping an eye of the interfaces you have on your machine and if there is a change is forces a restart of the apiserver (only the apiserver) so that k8s is available in all interfaces. If you do not need this behavior (eg you know the IP of your machine will not change or you want the apiserver on specific interfaces) you can bind the apiserver to a specific IP by adding the --advertise-address or --bind-address arguments in /var/snap/microk8s/current/args/kube-apiserver according to [2] and restart MicroK8s.

The behavior we have here causes all services to restart. It is as if there is a snap refresh happening but the latest/stable channel you are following has not changed for a while now. Could we see the out put of snap changes microk8s ?

Also you would better switch to channel 1.20/stable if this is a long lasting cluster. The following the latest/stable means that you will get upgraded to 1.21 when that comes out.

sudo snap refresh microk8s --channel=1.20/stable

I also see this snap-store snap installed. I do not think this would cause snap refreshes but if you do not need it you may want to remove it, just in case.

[1] https://github.com/ubuntu/microk8s/blob/master/microk8s-resources/wrappers/apiservice-kicker [2] https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/

[johngrabner]

The solution to keeping microk8s stable beyond a day is

• ubuntu UI “Software and Updates” tab 3 “Updates”, set to “never” … but dont forget to manual update
• sudo snap set system refresh.timer=sun
• disable wifi even if not used
• systemctl stop snap.microk8s.daemon-apiservice-kicker

In my case, will continue to investigate when time permits what one of the above caused the frequent resets to my app and update this ticket.