RuntimeError when trying to install Calico via YAML

Hello!

• Vote on this issue by adding a 👍 reaction
• To contribute a fix for this issue, leave a comment (and link to your pull request, if you’ve opened one already)

Issue details

Trying to install Calico CNI with Pulumi via YAML files fails with: RuntimeError: Set changed size during iteration

Steps to reproduce

1. Create a Python Pulumi program, trying to install Calico as instructed here: https://projectcalico.docs.tigera.io/getting-started/kubernetes/k3s/multi-node-install

"""A Kubernetes Python Pulumi program"""
import pulumi
from pulumi import ResourceOptions, Output

from pulumi_kubernetes.yaml import ConfigFile

calico_operator = ConfigFile(
    "calico-operator",
    file="https://projectcalico.docs.tigera.io/manifests/tigera-operator.yaml",
)

calico = ConfigFile(
    "calico",
    file="https://projectcalico.docs.tigera.io/manifests/custom-resources.yaml",
    opts=pulumi.ResourceOptions(depends_on=calico_operator)
)

1. Run pulumi preview
2. Watch the world burn, no really just Pulumi crashing 😦

Expected: Calico CNI is installed and Pulumi continues to install the rest of the stack.

Actual: Pulumi preview fails pulumi-preview.txt

Pulumi version: v3.24.1 Pulumi Python package: 3.24.1 pulumi-kubernetes version: 3.15.1

[AaronFriel]

@pasmon and @betamike thanks for your patience, getting back to this, it looks like there’s an issue with our Kubernetes provider and using “depends_on” with resources that have children. You’re likely to encounter this with ConfigFile, ConfigGroup, and Helm Charts.

A workaround is to use this in the resource options:

calico = ConfigFile(
    "calico",
    file="https://projectcalico.docs.tigera.io/manifests/custom-resources.yaml",
    opts=pulumi.ResourceOptions(
                 # ⬇️ parent resource 
        depends_on=calico_operator.resources.apply(lambda x: list(x.values())),
                                 # ⬆️ wait on all child resources 
    )
)

I also noticed that the Calico/Tigera operator manifest sets a field, “status”, we don’t allow in CRDs, so I’ve copied below a full working program with both workarounds:

"""A Kubernetes Python Pulumi program"""
from importlib.resources import Resource
from typing import Any, Dict, List
import pulumi
from pulumi import ResourceOptions, Output

from pulumi_kubernetes.yaml import ConfigFile

def _fix_status(obj: Any, opts: pulumi.ResourceOptions):
    if obj.get("kind") == "CustomResourceDefinition":
        if "status" in obj:
            # Remove status field from CRD, a field we do not permit setting on creating CRDs.
            del obj["status"]

calico_operator = ConfigFile(
    "calico-operator",
    file="https://projectcalico.docs.tigera.io/manifests/tigera-operator.yaml",
    transformations=[_fix_status]
)


calico = ConfigFile(
    "calico",
    file="https://projectcalico.docs.tigera.io/manifests/custom-resources.yaml",
    opts=pulumi.ResourceOptions(
        depends_on=calico_operator.resources.apply(lambda x: list(x.values())),
    )
)

[pasmon]

I bumped into this issue also with Kyverno ConfigFile, defining the Kyverno policies depending on the actual Kyverno installation. In this case, the depends_on workaround works for me. Thanks @AaronFriel !