[Bug] Operator continuously update resources in AKS

Describe the bug In our case we noticed that when we trying to create internal LB in AKS managed by Rancher (this is important) using annotation:

externalBootstrapService:
  metadata:
    annotations:
      service.beta.kubernetes.io/azure-load-balancer-internal: "true"
perPodService:
  metadata:
    annotations:
      service.beta.kubernetes.io/azure-load-balancer-internal: "true"

we see that time needed for creating LB is ~15 min. The root cause is that Strimzi operator and Rancher (which adds custom annotations like /metadata/annotations/field.cattle.io~1publicEndpoints) simultaneously try to change the same services of LoadBalancer type. Rancher adds some labels and annotations to services and Strimzi operator removes that data. That process caused continuously updating Azure load balancer configuration. As a result, Azure load balancer stays in an “updating” state with a long processing queue.

To Reproduce Steps to reproduce the behavior:

1. Create internal LB for external listener
2. Create AKS cluster managed by Rancher (https://rancher.com/docs/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters/aks/)
3. Create Internal LB by adding annotations: creating takes more than 15 min

Expected behavior LB creation takes less than 2 min.

Environment (please complete the following information):

• Strimzi version: 0.18
• Installation method: Helm chart
• Kubernetes cluster: Kubernetes 1.18
• Infrastructure: Azure AKS/Rancher

YAML files and logs

Additional context Suggestion: add ability ignore changes for annotations/labels for resources watched by operator:

example:
ignore_changes = [
      metadata[0].annotations["cattle.io/*"],
      metadata[0].annotations["field.cattle.io~1publicEndpoints"]
    ]

[scholzj]

@mluiten Strimzi does not remove the annotation per se. It just reconciles the resource => i.e. applies our version of it which removes these annotations because it does not have them. We do not have any whitelists or blacklists because so far it was never needed until this case. But for this particular usecase I would assume one can add it here: https://github.com/strimzi/strimzi-kafka-operator/blob/3e76f37c696bbb2712477eb541edb431bc8164fe/operator-common/src/main/java/io/strimzi/operator/common/operator/resource/ServiceOperator.java#L54

There we already handle some similar things in the service spec section such as assigned node ports or ipFamily etc. So I assume here we can have some allow-list for annotations which would be back ported from the original service to not get them deleted in the patch. That is at least where I planned to start … but of course contributions are always welcomed, so if you wanna look into it you are more then welcomed.

[mluiten]

@scholzj if you can point me in the right direction, I would be eager to take a look if I can try and make an improvement.

Why does Strimzi want to remove annotations that have nothing to do with Strimzi? Is there a whitelist of annotations it should care about?