Warning logs after updating to 1.23: etcd-client "retrying of unary invoker failed"

microk8s-inspect.tar.gz

Hi,

today I upgraded a staging 3-node cluster from 1.22 to 1.23. While checking the logs for kubelite, I noticed this message that keeps repeating

Jan 17 18:15:20 <hostname> microk8s.daemon-kubelite[94967]: {"level":"warn","ts":"2022-01-17T18:15:20.234Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000e9e1c0/#initially=[unix:///var/snap/microk8s/x3/var/kubernetes/backend/kine.sock:12379]","attempt":0,"error":"rpc error: code = Unimplemented desc = unknown service etcdserverpb.Maintenance"}

My cluster is running using dqlite, so I’m not entirely sure why I’m getting etcd-related logs. Is there something I can do to fix this?

[neoaggelos]

I can comment on this. The warning logs are indeed bening, this is the newer version of etcd-client in use by Kubernetes 1.23, which is running regular health-checks on the etcd endpoints.

The Kine version in use by MicroK8s (Kine is an etcd shim on top of dqlite) does not implement the particular API, hence the error. You can consider this error as being identical to a GET /health returning “404 Not Found”, because the health endpoint is not there.

Now, even though this is harmless at the moment, it is probably best to update our Kine version to implement this API. If anything, such errors are definitely noise and may be red-herrings during debugging other issues. Thanks @luca-nardelli for reporting the issue, this should be fixed soon.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.