Exclude secondary IP from produce_certs (and node service restarts)

Currently, MicroK8s is rebuilding the node certificates and restarting all the MicroK8s services on a node if any IP address changes. This conflicts with the operation of the PureLB Load Balancer when allocating local addresses.

When PureLB is installed and needs to assign an local IP address for load balancing use, it adds a secondary IP address to the ethernet adapter with a default route. This IP address change is triggering the daemon-apiserver-kicker to restart all the services. It then makes the running pods all go un-healthy so the pool IP allocation is removed and then reattempted on another node (or when this node comes back), and a never-ending cycle starts.

https://github.com/ubuntu/microk8s/blob/700359017d2c7f839638422e9d0ae876b4f7e632/microk8s-resources/wrappers/apiservice-kicker#L50-L68

Adding --advertise-address or --bind-address to args/kube-apiserver, while stopping the above process, seems to prevent a node join, I assume because the certificates do need to be rebuilt and the services restarted.

I propose that the fix is to adjust the get_ips() that produce_certs() uses to exclude any secondary IPs. This is how the function looks currently:

https://github.com/ubuntu/microk8s/blob/8d0b9e5d6cfdb61b855b0d1ad657b43c4db73583/microk8s-resources/actions/common/utils.sh#L330-L350

This is how the ip list looks on a node that PureLB’s lbnodeagent has allocated an ip address (10.3.50.193) based on a Load Balancer demand:

$ hostname -I
10.3.50.68 10.3.50.193 10.1.233.192
$ ip -o addr list
1: lo    inet 127.0.0.1/8 scope host lo\       valid_lft forever preferred_lft forever
1: lo    inet6 ::1/128 scope host \       valid_lft forever preferred_lft forever
2: eth0    inet 10.3.50.68/24 brd 10.3.50.255 scope global eth0\       valid_lft forever preferred_lft forever
2: eth0    inet 10.3.50.193/24 brd 10.3.50.255 scope global secondary eth0\       valid_lft forever preferred_lft forever
2: eth0    inet6 fe80::215:5dff:fe32:800/64 scope link \       valid_lft forever preferred_lft forever
6: vxlan.calico    inet 10.1.233.192/32 brd 10.1.233.192 scope global vxlan.calico\       valid_lft forever preferred_lft forever
6: vxlan.calico    inet6 fe80::641a:d3ff:fe93:5639/64 scope link \       valid_lft forever preferred_lft forever
7: calieeedeb80fed    inet6 fe80::ecee:eeff:feee:eeee/64 scope link \       valid_lft forever preferred_lft forever
16: kube-lb0    inet6 fe80::dc6c:83ff:fec8:c3df/64 scope link \       valid_lft forever preferred_lft forever

Are there any cases where the secondary ip addresses should be in the certificate?

hostname -I could return ipv6 addresses and PureLB can allocate ipv6 addresses, so my use of ip above includes both ipv4 and ipv6 instead of using -4.

Also, while I haven’t used it yet, PureLB can assign global virtual addresses to it’s kube-lb0 adapter, and those would need to be excluded from the reconfiguration as well.

cc: @adamdunstan as a PureLB developer.

[devZer0]

recent activity because stale bots suck

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.