CRD update fails if last-applied-configuration annotation too long

During testing, I noticed another strange behavior, if I should put it another issue, let me know. Looks like pulumi falsely states, that the prometheuses.monitoring.coreos.com CRD was updated when enableReplaceCRD is false, but actually, it wasn’t. I tested this on minikube with kubernetes version v1.21.2.

Steps to reproduce:

1. Make sure you’re using an older pulumi-kubernetes version than 3.12.1, I used 3.7.2.
2. Use the code from my first comment with enableReplaceCRD: false and deploy the chart version 18.0.7. It should be deployed fine and the prometheuses.monitoring.coreos.com CRD should have a long kubectl.kubernetes.io/last-applied-configuration annotation.
3. Save the CRD definition with kubectl get crd/prometheuses.monitoring.coreos.com -o yaml.
4. Update the chart to 23.3.2 and try to deploy, it should fail on prometheuses.monitoring.coreos.com with the message the Kubernetes API server reported that "prometheuses.monitoring.coreos.com" failed to fully initialize or become live: CustomResourceDefinition.apiextensions.k8s.io "prometheuses.monitoring.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes.
5. Update pulumi-kubernetes to 3.12.1.
6. Try to deploy again, it will show, that all CRDs were updated.
7. Save the CRD definition again and compare it with the previous version, it will be the same.
8. Optionally check pulumi state with pulumi stack export, it will show that prometheuses.monitoring.coreos.com CRD resource has the newest version, both in the inputs and outputs section. The easiest way to do this, just to check the controller-gen.kubebuilder.io/version annotation, for the 18.0.7 chart it will be v0.4.1, for 23.3.2 will show v0.6.2.
9. Set enableReplaceCRD: true then run pulumi up.
10. Run pulumi refresh, it will show, that ~__inputs changed on prometheuses.monitoring.coreos.com.
11. Optionally check pulumi state again, now both inputs and outputs will show the older CRD definition from the 18.0.7 chart.
12. Run pulumi up, prometheuses.monitoring.coreos.com will show an update, but will fail with the error: the Kubernetes API server reported that "prometheuses.monitoring.coreos.com" failed to fully initialize or become live: CustomResourceDefinition.apiextensions.k8s.io "prometheuses.monitoring.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes.

So with the latest pulumi-kubernetes version without enableReplaceCRD, it says that every CRD updated, even prometheuses.monitoring.coreos.com, but on the environment, it actually stays the same.

_Originally posted by @ncsibra in https://github.com/pulumi/pulumi-kubernetes/issues/1841#issuecomment-996760823_

[gitfool]

@lblackstone no unexpected changes now; thank you! 🎉

Resources:
    130 unchanged

[lblackstone]

Appreciate your patience on this fix! I got pulled off onto another project for the last several months and am just getting back to it. I’ve updated the PR, and should have a release out in the next day or two.