[v1.19.2] `helm install` brings 127.0.0.1:16443 not reachable

How it happens

Hello all. After I deploy microk8s using snap channel 1.19/stable(1.19/stable: v1.19.2 2020-09-26 (1710) 214MB classic), I experience helm chart install failure with the following error message

{"chart_repo": {"chart": {"name": "metallb", "ref": "bitnami/metallb", "version": "0.1.21"}, "config": {"configInline": {"address-pools": [{"addresses": ["172.16.0.0/16"], "avoid-buggy-ips": true, "name": "internal", "protocol": "layer2"}, {"addresses": ["10.101.2.50-10.101.2.100"], "avoid-buggy-ips": true, "name": "external", "protocol": "layer2"}]}}, "repository": {"name": "bitnami", "url": "https://charts.bitnami.com/bitnami"}}, "ansible_loop_var": "chart_repo", "changed": false, "command": "/snap/bin/helm --namespace=default --version=0.1.21 upgrade -i --reset-values -f=/tmp/tmpf7crurpc.yml metallb bitnami/metallb", "msg": "Failure when executing Helm command. Exited 1.\nstdout: Release \"metallb\" does not exist. Installing it now.\n\nstderr: Error: Kubernetes cluster unreachable: Get \"https://127.0.0.1:16443/version?timeout=32s\": dial tcp 127.0.0.1:16443: connect: connection refused\n", "stderr": "Error: Kubernetes cluster unreachable: Get \"https://127.0.0.1:16443/version?timeout=32s\": dial tcp 127.0.0.1:16443: connect: connection refused\n", "stderr_lines": ["Error: Kubernetes cluster unreachable: Get \"https://127.0.0.1:16443/version?timeout=32s\": dial tcp 127.0.0.1:16443: connect: connection refused"], "stdout": "Release \"metallb\" does not exist. Installing it now.\n", "stdout_lines": ["Release \"metallb\" does not exist. Installing it now."]}

The highlight of the above error log will be "Error: Kubernetes cluster unreachable: Get \"https://127.0.0.1:16443/version?timeout=32s\": dial tcp 127.0.0.1:16443: connect: connection refused", which indicated the cluster is not reachable in one of the node of the HA microk8s cluster. However, the cluster is running fine using kubectl get nodes on any node inside the HA microk8s cluster.

I had tried doing curl https://127.0.0.1:16443 on one of the nodes and the cluster returned 404 error arbitrarily mixed with normal unauthorized result (2 out of 10 times returned 404 on my trial).

My helm install is performed using Ansible module community.kubernetes.helm.

This issue is not happening on channel latest/stable (latest/stable: v1.19.0 2020-09-08 (1668) 214MB classic)

Many thanks.

Versions

Name	Version
Helm	v3.3.4
Ansible	2.9.12
Microk8s	v1.19.2

Report Tarball

inspection-report-20200928_112106.tar.gz

[ktsakalozos]

Indeed in there are a few fixes in the database layer waiting to be released in 1.19/candidate. However, before releasing to stable we need to first fix a memory leak identified in https://github.com/ubuntu/microk8s/issues/1598 .

[alwynpan]

This problem also appears when enable other add-ons such as storage, prometheus, and fluentd. Those add-ons can be enabled eventually after a few retries.

This error also pops out randomly: rpc error: code = Unknown desc = call query-sql (budget 0s): receive: header: EOF

When I watch a deployment by running watch microk8s.kubectl get pod -n some-namespace I can see those two errors appear randomly.