Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[BUG] Epicli upgrade issue - the process hangs for several hours on the task kubeadm upgrade apply

See original GitHub issue

Describe the bug Cannot upgrade the kubernetes cluster - looks like a random issue, most common for Azure configurations.

To Reproduce Steps to reproduce the behavior:

deploy 0.4.4 cluster
execute epicli upgrade (from develop branch)

Expected behavior The cluster has been successfully upgraded

OS (please complete the following information):

OS: [e.g. RHEL7.7, Ubuntu 18.04]

Cloud Environment (please complete the following information):

Cloud Provider [e.g. MS Azure]

Additional context


2020-06-29T22:04:48.1026877Z 22:04:48 INFO cli.engine.ansible.AnsibleCommand - TASK [upgrade : upgrade-master | Upgrade K8s cluster to v1.15.10 (using kubeadm-config.yml file)] ***
2020-06-29T22:49:14.7940500Z 22:49:14 INFO cli.engine.ansible.AnsibleCommand - FAILED - RETRYING: upgrade-master | Upgrade K8s cluster to v1.15.10 (using kubeadm-config.yml file) (20 retries left).
2020-06-29T23:37:26.7290995Z 23:37:26 INFO cli.engine.ansible.AnsibleCommand - FAILED - RETRYING: upgrade-master | Upgrade K8s cluster to v1.15.10 (using kubeadm-config.yml file) (19 retries left).
2020-06-29T23:42:59.1434418Z 23:42:59 INFO cli.engine.ansible.AnsibleCommand - FAILED - RETRYING: upgrade-master | Upgrade K8s cluster to v1.15.10 (using kubeadm-config.yml file) (18 retries left).
2020-06-30T00:31:11.0432338Z 00:31:11 INFO cli.engine.ansible.AnsibleCommand - FAILED - RETRYING: upgrade-master | Upgrade K8s cluster to v1.15.10 (using kubeadm-config.yml file) (17 retries left).
2020-06-30T00:36:43.4952279Z 00:36:43 INFO cli.engine.ansible.AnsibleCommand - FAILED - RETRYING: upgrade-master | Upgrade K8s cluster to v1.15.10 (using kubeadm-config.yml file) (16 retries left).
2020-06-30T01:24:55.4254331Z 01:24:55 INFO cli.engine.ansible.AnsibleCommand - FAILED - RETRYING: upgrade-master | Upgrade K8s cluster to v1.15.10 (using kubeadm-config.yml file) (15 retries left).
2020-06-30T01:30:27.8367170Z 01:30:27 INFO cli.engine.ansible.AnsibleCommand - FAILED - RETRYING: upgrade-master | Upgrade K8s cluster to v1.15.10 (using kubeadm-config.yml file) (14 retries left).
2020-06-30T02:18:39.8409973Z 02:18:39 INFO cli.engine.ansible.AnsibleCommand - FAILED - RETRYING: upgrade-master | Upgrade K8s cluster to v1.15.10 (using kubeadm-config.yml file) (13 retries left).
2020-06-30T02:24:12.2573132Z 02:24:12 INFO cli.engine.ansible.AnsibleCommand - FAILED - RETRYING: upgrade-master | Upgrade K8s cluster to v1.15.10 (using kubeadm-config.yml file) (12 retries left).
2020-06-30T03:12:24.2749881Z 03:12:24 INFO cli.engine.ansible.AnsibleCommand - FAILED - RETRYING: upgrade-master | Upgrade K8s cluster to v1.15.10 (using kubeadm-config.yml file) (11 retries left).
2020-06-30T03:17:56.7150480Z 03:17:56 INFO cli.engine.ansible.AnsibleCommand - FAILED - RETRYING: upgrade-master | Upgrade K8s cluster to v1.15.10 (using kubeadm-config.yml file) (10 retries left).
2020-06-30T04:06:08.6468293Z 04:06:08 INFO cli.engine.ansible.AnsibleCommand - FAILED - RETRYING: upgrade-master | Upgrade K8s cluster to v1.15.10 (using kubeadm-config.yml file) (9 retries left).
2020-06-30T04:11:41.0625442Z 04:11:41 INFO cli.engine.ansible.AnsibleCommand - FAILED - RETRYING: upgrade-master | Upgrade K8s cluster to v1.15.10 (using kubeadm-config.yml file) (8 retries left).
2020-06-30T04:59:53.0070741Z 04:59:53 INFO cli.engine.ansible.AnsibleCommand - FAILED - RETRYING: upgrade-master | Upgrade K8s cluster to v1.15.10 (using kubeadm-config.yml file) (7 retries left).
2020-06-30T05:05:25.4451723Z 05:05:25 INFO cli.engine.ansible.AnsibleCommand - FAILED - RETRYING: upgrade-master | Upgrade K8s cluster to v1.15.10 (using kubeadm-config.yml file) (6 retries left).
2020-06-30T05:53:37.9468084Z 05:53:37 INFO cli.engine.ansible.AnsibleCommand - FAILED - RETRYING: upgrade-master | Upgrade K8s cluster to v1.15.10 (using kubeadm-config.yml file) (5 retries left).
2020-06-30T05:59:09.7340515Z 05:59:09 INFO cli.engine.ansible.AnsibleCommand - FAILED - RETRYING: upgrade-master | Upgrade K8s cluster to v1.15.10 (using kubeadm-config.yml file) (4 retries left).

E0629 11:49:57.067369       1 reflector.go:125] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to list *v1.Secret: illegal base64 data at input byte 3
E0629 11:49:58.070606       1 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Secret: illegal base64 data at input byte 3
E0629 11:49:58.072162       1 reflector.go:125] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to list *v1.Secret: illegal base64 data at input byte 3
E0629 11:49:59.075549       1 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Secret: illegal base64 data at input byte 3
E0629 11:49:59.075968       1 reflector.go:125] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to list *v1.Secret: illegal base64 data at input byte 3
E0629 11:50:00.089012       1 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Secret: illegal base64 data at input byte 3
E0629 11:50:00.091010       1 reflector.go:125] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to list *v1.Secret: illegal base64 data at input byte 3
E0629 11:50:01.093364       1 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Secret: illegal base64 data at input byte 3

Looks like an issue with etcd encryption.

Issue Analytics

State:
Created 3 years ago
Comments:5 (3 by maintainers)

Top GitHub Comments

1reaction

przemyslaviccommented, Jul 15, 2020

@mkyc I have recently run other tests on the same pipelines and the AWS/RedHat environments failed due to exceeding the VPC limit. But that was after testing this fix. On Azure DevOps we have results from the previous run. Only 2 configurations out of 30 partially succeeded (due to problems with the RabbitMQ deployment, which is not part of this task), all others were successful.

0reactions

mkyccommented, Jul 15, 2020

@przemyslavic how is that we have “failed” label in “develop” section of almost all types of tests? https://github.com/epiphany-platform/epiphany/blob/develop/docs/home/TESTING.md

Top Results From Across the Web

kubeadm upgrade plan and kubeadm upgrade apply v1.10.1 ...

Similarly, kubeadm upgrade apply v1.10.1 hangs before changing any manifests (control plane pods don't restart at all). Output before hanging:.

Upgrading kubeadm clusters - Kubernetes

The upgrade procedure on worker nodes should be executed one node at a time or few nodes at a time, without compromising the...

Issue with Practice Test Cluster Upgrade Process - KodeKloud

Error from validation: Task not completed: Master Upgraded to v1.18.0 ... you have upgraded the control plane with 'kubeadm upgrade apply':

Upgrades - Rancher Docs

This page covers the following topics: How upgrades work; Prerequisites; Upgrading Kubernetes; Configuring the upgrade strategy; Maintaining availability for ...

How To Upgrade Kubernetes Cluster Using Kubeadm?

Apply the upgrade using the following command. sudo kubeadm upgrade node. Now if you check the kubeadm version, you can see the upgraded...