strimz-kafka-operator restarting kafka due to fluxcd

Describe the bug We started using strimz-kafka-operator recently and it was overall smooth (thank you!).

We ran into some problems with fluxcd (The GitOps tool we use https://fluxcd.io/): strimz-kafka-operator would restart kafka sometimes when we push a change. Looking a bit into it and it seems that some lables added by fluxcd, triggered strimz-kafka-operator to do a rolling update.

Expected behavior strimzi-kafka-operator ignore all fluxcd tags.

Environment (please complete the following information):

• Strimzi version: main
• Installation method: YAML files via fluxcd
• Kubernetes cluster: Kubernetes 1.20
• Infrastructure: Amazon EKS

YAML files and logs

This is the log from the operator: https://gist.github.com/kangzhang/13cec69ce30079b94d4f3bf41631d6a6 . You can see that the operator detected some label changes by fluxcd: https://gist.github.com/kangzhang/13cec69ce30079b94d4f3bf41631d6a6#file-gistfile1-txt-L244 and triggered a rolling update: https://gist.github.com/kangzhang/13cec69ce30079b94d4f3bf41631d6a6#file-gistfile1-txt-L267

Additional context

I’m wondering if we could black list fluxcd labels? The are only used by fluxcd to track the deployment and should not be used by strimzi.

Here are some example labels fluxcd would add:

              kustomize.toolkit.fluxcd.io/checksum=77dd8897619d50e2035bafbfe0b41e22cea893cd
              kustomize.toolkit.fluxcd.io/name=environment
              kustomize.toolkit.fluxcd.io/namespace=flux-system

I also read thru #2558 and #5083 . The flag STRIMZI_LABELS_EXCLUSION_PATTERN does not seem to help. (I already set to "(^app.kubernetes.io/(?!part-of).*|^kustomize.toolkit.fluxcd.io.*)" and you can see it in log file too: https://gist.github.com/kangzhang/13cec69ce30079b94d4f3bf41631d6a6#file-gistfile1-txt-L69

Thanks!

[scholzj]

FYI: I opened #5245 to exclude these by default.

[nicknotfun]

The main labels are a by-product of garbage collection, listed here: https://fluxcd.io/docs/components/kustomize/kustomization/#garbage-collection

The Flux process is on reconciling a change to the source repository it marks any resources with these labels, so that if later the item is removed from the source, it knows which items to GC