[Bug] unexpected removing all kafka resources when upgrade using helm3

Describe the bug I’m using strimzi operator v0.19.0 and tried upgrade to 0.20.0. When I’ve ran helm upgrade procedure all my resources (users, topics, clusters) was removed. I try to reproduce problem with fresh installed cluster and situation was reproduced again.

To Reproduce Steps to reproduce the behavior:

1. helm install strimzi-kafka strimzi/strimzi-kafka-operator --namespace kafka --set watchNamespaces="{kafka,test-kafka}" --version=0.19.0
2. create cluster, users and topics from manifests (apiVersion: v1beta1)
3. helm upgrade strimzi-kafka strimzi/strimzi-kafka-operator --namespace kafka --set watchNamespaces="{kafka,test-kafka}"
kubectl get crd| grep kafka| wc -l
       0

After the steps above my cluster and users/topics was removed. The operator pod try to start and crashed with the following error:

2020-10-26 14:35:47 WARN  WatchConnectionManager:198 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found'
	at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
2020-10-26 14:35:47 WARN  WatchConnectionManager:198 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found'
	at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]

Expected behavior The operator should be updated without removing resources.

Environment (please complete the following information):

• Strimzi version: 0.19.0
• Installation method: Helm chart
• Kubernetes cluster: v.1.18.8
• Infrastructure: Rancher2 on Amazon EC2 instances

[gris-gris]

A quick workaround we found with our team:

1. Backup and delete all secrets related to Strimzi helm chart sh.helm.release.v1.strimzi
2. Redeploy with version 0.20.0, CRD’s will not be deleted and version upgrade completes successfully

[driosalido]

A quick workaround we found with our team:

1. Backup and delete all secrets related to Strimzi helm chart sh.helm.release.v1.strimzi
2. Redeploy with version 0.20.0, CRD’s will not be deleted and version upgrade completes successfully

There is also the option to edit the data in the helm secret instead of deleting it

kubectl get secrets -n NAMESPACE sh.helm.release.v1.DEPLOYNAME -o json | jq .data.release -r | base64 --decode | base64 --decode | gunzip - > /var/tmp/manifest.json

The remove the CRD data inside the templates and manifest sections and upload the secret again

DATA=`cat /var/tmp/manifest.json | gzip -c | base64 | base64`
kubectl patch secret -n NAMESPACE sh.helm.release.v1.DEPLOYNAME --type='json' -p="[ {\"op\":\"replace\",\"path\":\"/data/release\",\"value\":\"$DATA\"}]"

Then the upgrade to 0.20.0 will leave the CRDs alone…