Using Class in Ingress Causes Error

Describe the bug Trying to use nginx ingress and receiving error from the operator.

To Reproduce Steps to reproduce the behavior:

1. Create a listener of type ingress
2. Add configuration.class value of nginx
3. Apply
4. See error in operator

2021-03-30 21:27:49 WARN AbstractOperator:508 - Reconciliation #32(watch) Kafka(strimzi/company): Failed to reconcile
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://10.0.0.1/apis/networking.k8s.io/v1/namespaces/strimzi/ingresses. Message: Ingress.extensions "company-kafka-external-bootstrap" is invalid: annotations.kubernetes.io/ingress.class: Invalid value: "nginx": can not be set when the class field is also set. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=annotations.kubernetes.io/ingress.class, message=Invalid value: "nginx": can not be set when the class field is also set, reason=FieldValueInvalid, additionalProperties={})], group=extensions, kind=Ingress, name=company-kafka-external-bootstrap, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Ingress.extensions "company-kafka-external-bootstrap" is invalid: annotations.kubernetes.io/ingress.class: Invalid value: "nginx": can not be set when the class field is also set, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:570) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:509) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:474) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:435) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:250) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:871) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:366) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:85) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
at io.strimzi.operator.common.operator.resource.AbstractResourceOperator.internalCreate(AbstractResourceOperator.java:213) ~[io.strimzi.operator-common-0.22.1.jar:0.22.1]
at io.strimzi.operator.common.operator.resource.AbstractResourceOperator.lambda$reconcile$0(AbstractResourceOperator.java:104) ~[io.strimzi.operator-common-0.22.1.jar:0.22.1]
at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
at java.lang.Thread.run(Thread.java:834) [?:?]

Expected behavior The ingress should be created with the proper kube annotation for ingressClassName.

Environment (please complete the following information):

• Strimzi version: 0.22.1
• Installation method: Helm chart from https://strimzi.io/charts/
• Kubernetes cluster: Kubernetes 1.19
• Infrastructure: Azure AKS

YAML files and logs

      - name: external
        port: 9096
        type: ingress
        tls: true
        configuration:
          class: nginx
          brokerCertChainAndKey:
            secretName: kubeingress-tls
            certificate: tls.crt
            key: tls.key
          bootstrap:
            host: kafka-bootstrap.test.company.com
          brokers:
            - broker: 0
              host: kafka-0.test.company.com
            - broker: 1
              host: kafka-1.test.company.com
            - broker: 2
              host: kafka-2.test.company.com
        authentication:
          type: scram-sha-512

Additional context I’d really love to use Istio (as standard Kube Ingress Class) rather than nginx, but appears to be very specifically coded for nginx.

[treyhendon]

Update - I was able to work-around this by using the annotation feature for the listener.

      - name: external
        port: 9096
        type: ingress
        tls: true
        configuration:
          bootstrap:
            host: kafka-bootstrap.{{ .Values.strimzi.tld }}
            annotations:
              kubernetes.io/ingress.class: istio
          brokers:
            - broker: 0
              host: kafka-0.test.company.com
              annotations:
                kubernetes.io/ingress.class: istio
            - broker: 1
              host: kafka-1.test.company.com
              annotations:
                kubernetes.io/ingress.class: istio
            - broker: 2
              host: kafka-2.test.company.com
              annotations:
                kubernetes.io/ingress.class: istio
        authentication:
          type: scram-sha-512

Additional note, I do not actually have a Kube IngressClass object created in the cluster.

[scholzj]

Sorry, I was busy with something else.

It looks like I managed to reproduce it finally in my environment … it seems that:

• When you modify an existing Kubernetes resource, Kubernetes does not care about having both the kubernetes.io/ingress.class annotation as well as spec.ingressClassName.
• When you try to create a new Ingress with both specified, it complains.

So when you have the Kafka cluster deployed first without having the class: abc in the Kafka CR and then add it later, Kube is happy and doesn’t care about it. But when you set the class in the Kafka CR right from the beginning, it will reject it.

I’m not completely happy about it … but I guess removing the annotation and having people set it manually if needed is the only real option to go forward. I will open a PR.