Pulumi up/preview fails on conflict for Helm-installed Istio

What happened?

We’ve been using the below to install Istio into an EKS cluster (k8s 1.22). It’s been fine so far.

istio_base = k8s.helm.v3.Chart('istio-base',
                               ChartOpts(
                                   fetch_opts=FetchOpts(
                                       repo='https://istio-release.storage.googleapis.com/charts'
                                   ),
                                   namespace="istio-system",
                                   chart='base',
                               ),
                               opts=pulumi.ResourceOptions(
                                   providers={
                                       'kubernetes': provider,
                                   },
                                   depends_on=istio_ns
                               ))

Things have been great for a month. Today, preview and up fail with the following error without any updates to the Pulumi project containing this code.

        ~ kubernetes:admissionregistration.k8s.io/v1:ValidatingWebhookConfiguration: (update)
            [id=istiod-default-validator]
            [urn=urn:pulumi:dev::services::kubernetes:helm.sh/v3:Chart$kubernetes:admissionregistration.k8s.io/v1:ValidatingWebhookConfiguration::istiod-default-validator]
            [provider=urn:pulumi:dev::services::pulumi:providers:kubernetes::provider::ca67d70d-1289-478d-8b60-ddc901173fef]
          ~ webhooks: [
              ~ [0]: {
                      ~ failurePolicy: "Fail" => "Ignore"
                    }
            ]
error: Preview failed: 1 error occurred:
	* the Kubernetes API server reported that "istiod-default-validator" failed to fully initialize or become live: use `pulumi.com/patchForce` to override the conflict: Apply failed with 2 conflicts: conflicts with "pilot-discovery" using admissionregistration.k8s.io/v1:
- .webhooks[name="validation.istio.io"].failurePolicy
conflicts with "pulumi-resource-kubernetes" using admissionregistration.k8s.io/v1:
- .webhooks[name="validation.istio.io"].rules

If I look at the ValidatingWebhookConfiguration on the cluster, it is set to failurePolicy: Fail. If I look at the project’s current stack, I can see that the Input was set to Ignore on urn:pulumi:dev::services::kubernetes:helm.sh/v3:Chart$kubernetes:admissionregistration.k8s.io/v1:ValidatingWebhookConfiguration::istiod-default-validator

                "inputs": {
                    "apiVersion": "admissionregistration.k8s.io/v1",
                    "kind": "ValidatingWebhookConfiguration",
....
                    "webhooks": [
...
                            "failurePolicy": "Ignore",

But, the Outputs have the Fail which match the state on K8s.

                "outputs": {
...
                    "apiVersion": "admissionregistration.k8s.io/v1",
                    "kind": "ValidatingWebhookConfiguration",
...
                    "webhooks": [
...
                            "failurePolicy": "Fail",

My understanding is that Istio itself will change the failurePolicy on this resource from Ignore to Fail after it has initialized itself – that’s fine. However, I don’t know why Pulumi has suddenly started failing to apply future changes, nor how to convince Pulumi that everything is “OK”.

Steps to reproduce

Code below is unchanged. Previously worked fine.

istio_base = k8s.helm.v3.Chart('istio-base',
                               ChartOpts(
                                   fetch_opts=FetchOpts(
                                       repo='https://istio-release.storage.googleapis.com/charts'
                                   ),
                                   namespace="istio-system",
                                   chart='base',
                               ),
                               opts=pulumi.ResourceOptions(
                                   providers={
                                       'kubernetes': provider,
                                   },
                                   depends_on=istio_ns
                               ))

Expected Behavior

Expected pulumi preview/pulumi up to be capable of reconciling the state from Kubernetes and Pulumi operations to succeed.

Actual Behavior

Failure to pulumi preview and pulumi up with the following

  kubernetes:helm.sh/v3:Chart$kubernetes:admissionregistration.k8s.io/v1:ValidatingWebhookConfiguration (istiod-default-validator)
    error: 1 error occurred:
	* the Kubernetes API server reported that "istiod-default-validator" failed to fully initialize or become live: use `pulumi.com/patchForce` to override the conflict: Apply failed with 2 conflicts: conflicts with "pilot-discovery" using admissionregistration.k8s.io/v1:
- .webhooks[name="validation.istio.io"].failurePolicy
conflicts with "pulumi-resource-kubernetes" using admissionregistration.k8s.io/v1:
- .webhooks[name="validation.istio.io"].rules

Output of pulumi about

CLI
Version      3.43.1
Go Version   go1.19.2
Go Compiler  gc

Plugins
NAME        VERSION
aws         5.13.0
awsx        1.0.0-beta.10
docker      3.4.1
eks         0.41.2
kafka       3.3.0
kubernetes  3.22.0
python      unknown
random      4.8.2

Host
OS       darwin
Version  12.6
Arch     arm64

This project is written in python: executable='/Users/elserj/.pyenv/shims/python3' version='3.9.13

Additional context

Tried to run a pulumi refresh, thinking that Pulumi’s state was expecting the original Ignore value for this field. This refresh appears to have had no effect.

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you’ve opened one already).

[lblackstone]

It was a frustrating event to run head-first into on a Friday night

Sorry about that; we intended to release earlier in the week, but ran into some unexpected issues that pushed it back. We generally try to avoid Friday releases, but had a deadline in this case.

Not sure if there’s a concise documentation/warning you could even make, but maybe this thread will help others who may also run head-first into this.

We’re working on this today across multiple channels. Thanks for the report, and the quick responses!

[joshelser]

For this particular case, you should be able to work around that field changing by setting the ignoreChanges resource option for the ValidatingWebhookConfiguration resource. Pulumi is flagging a legitimate difference that wasn’t apparent with a Client-Side preview, but in this case, you want to explicitly ignore that difference.

Thanks for the pointer, Levi! For now, we’ve just disabled it (don’t have the cycles to switch over from client side to server side), but I’ll leave your suggestion for us to return to.

I don’t think there’s any actual bug here to “fix”. It was a frustrating event to run head-first into on a Friday night, but it was the sum of multiple things (e.g. we didn’t pin pulumi-kubernetes, we didn’t explicitly disable SSA, the object we were deploying was subject to the change in state as you said). Not sure if there’s a concise documentation/warning you could even make, but maybe this thread will help others who may also run head-first into this. 👍🏼 by me to close this.