Helm chart v0.22.1 failed to upgrade Strimzi operator to support v1beta2 API

Please use this to only for bug reports. For questions or when you need help, you can use the GitHub Discussions, our #strimzi Slack channel or out user mailing list.

Describe the bug Upgraded Strimzi operator from 0.21.1 to 0.22.1 Then follow the instrucstion in 7.2.2. Converting custom resources configuration files using the API conversion tool to convert CRD from v1beta1 to v1beta2

To Reproduce Steps to reproduce the behavior:

Current Strimzi operator is at version 0.21.1. Upgrade to latest version by

helm repo update
helm upgrade nystrimzi strimzi/strimzi-kafka-operator --namespace myns

Then use bin/api-conversion.sh downloaded from release, api-conversion-0.22.1.zip

• bin/api-conversion.sh convert-file : works OK
• bin/api-conversion.sh convert-resource --all-namespaces: works OK

In spite of the seemingly successful output , such as “Kafka resource named myexistingcluster1 in namespace kafka has been converted”. The existing resources seems NOT to be converted. They still show apiVersion: kafka.strimzi.io/v1beta1

kubectl get kafka/myexistingcluster1 -n kafka -o yaml

The step “Upgrading CRDs to v1beta2” failed"

bin/api-conversion.sh crd-upgrade -d

Changing stored version in all Strimzi CRDs to v1beta2:
Updating KafkaConnector CRD

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://MYCLUSTER.azmk8s.io/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/kafkausers.kafka.strimzi.io.
 Message: CustomResourceDefinition.apiextensions.k8s.io "kafkausers.kafka.strimzi.io" is invalid: spec.versions: Invalid
 value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:"v1beta1", Se
rved:true, Storage:false, Deprecated:false, DeprecationWarning:(*string)(nil), Schema:(*apiextensions.CustomResourceVali
dation)(nil), Subresources:(*apiextensions.CustomResourceSubresources)(nil), AdditionalPrinterColumns:[]apiextensions.Cu
stomResourceColumnDefinition(nil)}, apiextensions.CustomResourceDefinitionVersion{Name:"v1alpha1", Served:true, Storage:
false, Deprecated:false, DeprecationWarning:(*string)(nil), Schema:(*apiextensions.CustomResourceValidation)(nil), Subre
sources:(*apiextensions.CustomResourceSubresources)(nil), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnD
efinition(nil)}}: must have exactly one version marked as storage version. Received status: Status(apiVersion=v1, code=4
22, details=StatusDetails(causes=[StatusCause(field=spec.versions, message=Invalid value: []apiextensions.CustomResource
DefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:"v1beta1", Served:true, Storage:false, Deprecated:f
alse, DeprecationWarning:(*string)(nil), Schema:(*apiextensions.CustomResourceValidation)(nil), Subresources:(*apiextens
ions.CustomResourceSubresources)(nil), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition(nil)}, ap
iextensions.CustomResourceDefinitionVersion{Name:"v1alpha1", Served:true, Storage:false, Deprecated:false, DeprecationWa
rning:(*string)(nil), Schema:(*apiextensions.CustomResourceValidation)(nil), Subresources:(*apiextensions.CustomResource
Subresources)(nil), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition(nil)}}: must have exactly on
e version marked as storage version, reason=FieldValueInvalid, additionalProperties={})], group=apiextensions.k8s.io, ki
nd=CustomResourceDefinition, name=kafkausers.kafka.strimzi.io, retryAfterSeconds=null, uid=null, additionalProperties={}
), kind=Status, message=CustomResourceDefinition.apiextensions.k8s.io "kafkausers.kafka.strimzi.io" is invalid: spec.ver
sions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name
:"v1beta1", Served:true, Storage:false, Deprecated:false, DeprecationWarning:(*string)(nil), Schema:(*apiextensions.Cust
omResourceValidation)(nil), Subresources:(*apiextensions.CustomResourceSubresources)(nil), AdditionalPrinterColumns:[]ap
iextensions.CustomResourceColumnDefinition(nil)}, apiextensions.CustomResourceDefinitionVersion{Name:"v1alpha1", Served:
true, Storage:false, Deprecated:false, DeprecationWarning:(*string)(nil), Schema:(*apiextensions.CustomResourceValidatio
n)(nil), Subresources:(*apiextensions.CustomResourceSubresources)(nil), AdditionalPrinterColumns:[]apiextensions.CustomR
esourceColumnDefinition(nil)}}: must have exactly one version marked as storage version, metadata=ListMeta(_continue=nul
l, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failur
e, additionalProperties={}).

Keep repeating bin/api-conversion.sh crd-upgrade -d seems to see progress, each time the output involves another CRD

Changing stored version in all Strimzi CRDs to v1beta2:                                                                 
Updating v1beta1 version of Kind KafkaUser to be served but not stored                                                  
Updating v1alpha1 version of Kind KafkaUser to be served but not stored                                                 
Updating KafkaUser CRD                                                                                                  
... skip exception msg ...

Changing stored version in all Strimzi CRDs to v1beta2:
Updating v1alpha1 version of Kind KafkaBridge to be served but not stored
Updating KafkaBridge CRD
... skip exception msg ...

Changing stored version in all Strimzi CRDs to v1beta2:
Updating v1alpha1 version of Kind KafkaConnector to be served but not stored
Updating KafkaConnector CRD
... skip exception msg ...

Changing stored version in all Strimzi CRDs to v1beta2:
Updating v1beta1 version of Kind KafkaTopic to be served but not stored
Updating v1alpha1 version of Kind KafkaTopic to be served but not stored
Updating KafkaTopic CRD
... skip exception msg ...

Changing stored version in all Strimzi CRDs to v1beta2:
Updating v1alpha1 version of Kind KafkaRebalance to be served but not stored
Updating KafkaRebalance CRD
... skip exception msg ...

Changing stored version in all Strimzi CRDs to v1beta2:                                                                 
Updating v1beta1 version of Kind Kafka to be served but not stored                                                      
Updating v1alpha1 version of Kind Kafka to be served but not stored                                                     
Updating Kafka CRD                                                                                                      
... skip exception msg ...

But the exceptions seems to be persisitent as the same failed output keeps repeating. Attempt to create a Kafka cluster with apiVersion: kafka.strimzi.io/v1beta2 failed with error:

kubectl apply -f deploy/mykafka.yaml
error: unable to recognize "deploy/mykafka.yaml": no matches for kind "Kafka" in version "kafka.strimzi.io/v1beta2"

Expected behavior

bin/api-conversion.sh crd-upgrade -d should succeed in one go. And allow to create resource with kafka.strimzi.io/v1beta2 API version

Environment (please complete the following information):

• Strimzi version: 0.2.1 upgraded to 0.22.1
• Installation method: Helm chart
• Kubernetes cluster: Kubernetes version 1.19.7
• Infrastructure: Azure AKS

YAML files and logs

The deploy/mykafka.yaml which had failed to be created

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kkbugreport
  namespace: tridebug
spec:
  kafka:
    version: 2.7.0
    replicas: 1
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
        #authentication:
        #  type: tls
      - name: external
        port: 9094
        type: loadbalancer
        tls: true
        #authentication:
        #  type: tls
        configuration:

    config:
      auto.create.topics.enable: false
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      log.message.format.version: "2.7"
    storage:
      deleteClaim: false
      size: 1Gi
      type: persistent-claim
  zookeeper:
    replicas: 1
    livenessProbe:
      initialDelaySeconds: 60
      timeoutSeconds: 5
    readinessProbe:
      initialDelaySeconds: 60
      timeoutSeconds: 5
    resources: {}
    storage:
      deleteClaim: false
      size: 1Gi
      type: persistent-claim

  entityOperator:
    topicOperator: {}
    userOperator: {}

Additional context Add any other context about the problem here.

[tringuyen-yw]

Thank you very much for your help @scholzj Confirmed the manual upgrade to 0.22.1 works. This must have been a flaw in the helm chart v0.22.1 which didn’t update the physical strimzi operators. This issue is probably relevant I am going to edit the title to mention that is rather a helm chart bug.

Here is the manual procedure in case this helps someone having the same troubles to upgrade the Strimzi api to the new version v1beta2:

cd ~/Downloads
wget https://github.com/strimzi/strimzi-kafka-operator/releases/download/0.22.1/strimzi-0.22.1.tar.gz
tar xzvf strimzi-0.22.1.tar.gz
cd ./strimzi-0.22.1/install

kubectl replace -f ./cluster-operator/
kubectl replace -f ./strimzi-admin/
kubectl replace -f ./topic-operator/
kubectl replace -f ./user-operator/

# is there `v1beta2` support?
kubectl get crd kafkas.kafka.strimzi.io -o jsonpath="{.spec.versions[*].name}{'\n'}"
v1beta2 v1beta1 v1alpha1

wget https://github.com/strimzi/strimzi-kafka-operator/releases/download/0.22.1/api-conversion-0.22.1.tar.gz
tar xzvf api-conversion-0.22.1.tar.gz
cd ./api-conversion-0.22.1

# convert existing Strimzi managed resources to `v1beta2`
bin/api-conversion.sh convert-resource --all-namespaces

# Upgrading CRDs to v1beta2
bin/api-conversion.sh crd-upgrade --debug

[scholzj]

Ahh, ok … well, as far as I understood, Helm 3 does not upgrade the CRDs anymore. So you basically need to do it manually. For more info see https://strimzi.io/docs/operators/latest/full/deploying.html#upgrading_the_cluster_operator_using_helm_chart or https://helm.sh/docs/chart_best_practices/custom_resource_definitions/