"pulumi preview" needs 'patch' permission when SSA is enabled

What happened?

Thanks to https://github.com/pulumi/pulumi-eks/issues/800 we got a preview of what enabling SSA via pulumi-k8s 3.22.0 does to our infra-as-code config (we pin 3.21.4):

• We run pulumi preview with kubernetes credentials that only allow read, list, watch of all resources (never create/modify).
• The pulumi-eks provider creates a pulumi-kubernetes provider at version 3.22.0, with default values - enabling SSA
• When SSA is enabled, it attempts to modify the aws-auth configmap in the cluster.

This results in the following error message:

    kubernetes:core/v1:ConfigMap mzcloud-staging-system-nodeAccess  error: configmaps "aws-auth" is forbidden: User "gha-readonly-oidc:GithubActionsSession-pulumi-preview" cannot patch resource "configmaps" in API group "" in the namespace "kube-system"

Steps to reproduce

1. create a cluster role that’s similar to the following code:

          ro_role = k8s.rbac.v1.ClusterRole(
              f"{name}-gha-readonly-access",
              metadata=k8s.meta.v1.ObjectMetaArgs(
                  name="gha-readonly-access",
              ),
              rules=[
                  k8s.rbac.v1.PolicyRuleArgs(
                      api_groups=["*"],
                      resources=["*"],
                      verbs=["get", "list", "watch"],
                  )
              ],
              opts=pulumi.ResourceOptions(
                  provider=self.provider,
                  parent=self,
              ),
          )
          ro_group_name = "gha-readonly-access"
          k8s.rbac.v1.ClusterRoleBinding(
              f"{name}-gha-readonly-access",
              metadata=k8s.meta.v1.ObjectMetaArgs(name="gha-readonly-access"),
              subjects=[k8s.rbac.v1.SubjectArgs(kind="Group", name=ro_group_name)],
              role_ref=k8s.rbac.v1.RoleRefArgs(
                  kind="ClusterRole",
                  name=ro_role.metadata.name,
                  api_group="rbac.authorization.k8s.io",
              ),
              opts=pulumi.ResourceOptions(
                  provider=self.provider,
                  parent=self,
              ),
          )
          role_mappings_list.append(
              {
                  "rolearn": stack_config.GHA_READONLY_ROLE,
                  "username": r"gha-readonly-oidc:{{SessionName}}",
                  "groups": [ro_group_name],
              },
          )

1. Configure the kubernetes credentials to use that role to authenticate into the EKS cluster
2. Use pulumi-eks to define an EKS cluster with a given role/authorization mapping
3. Run pulumi preview with those read-only credentials
4. Observe the error above.

Expected Behavior

pulumi preview should never attempt to modify any data.

Actual Behavior

pulumi preview attempts to modify data and is (thankfully!) not permitted to.

Output of pulumi about

CLI
Version      3.42.0
Go Version   go1.19.1
Go Compiler  gc

Plugins
NAME              VERSION
aws               5.16.0
cloudinit         1.3.0
command           0.5.2
crds              0.0.0
docker-buildkit   0.1.20
eks               0.40.0
fivetran          0.1.6
frontegg          0.2.24
honeycomb         0.0.13
kubernetes        3.21.4
kubernetes-proxy  0.1.3
linkerd-link      0.0.7
postgresql        3.6.0
postgresql-exec   0.1.1
python            unknown
random            4.8.2
tls               4.6.1

Host
OS       nixos
Version  22.05 (Quokka)
Arch     x86_64

This project is written in python: executable='/home/asf/Hacks/cloud/venv/bin/python3' version='3.9.13

(redacted stack and list of resources)

Additional context

We’re attempting to “pin” the pulumi-kubernetes provider at 3.21.4 while we figure out how to live without SSA for the time being - ineffectively. I’m going to post about our problems in https://github.com/pulumi/pulumi/issues/10758 to hopefully get us a solution for opting into our own upgrade cadence.

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you’ve opened one already).

[antifuchs]

I just double-checked the stack configuration locally and confirmed that the setting you indicated disables SSA mode for me. We’ll need to do some more investigation to see if we can find out why that didn’t work for you.

After discussion with @ringods, it sounds like this is due to the way pulumi-eks interacts with pulumi-kubernetes: It disregards the kubernetes:enableServerSideApply setting for the provider that it creates. The setting is correct for all the k8s providers that we create in our code directly, just not in the pulumi-eks created resources.

[lblackstone]

I spent some more time reading through the upstream discussion about this, and it looks like patch permission is going to be a hard requirement of this feature. There were a few other users pushing for a way to filter previews by verb, but those discussions didn’t go anywhere as far as I can tell.

Given that information, I see the following alternatives:

1. Add/retain a client-side preview mode that doesn’t use dry-run
2. Use an admission controller on the cluster to restrict mutating operations

Unfortunately, both of these approaches have significant drawbacks. Fundamentally, the only way to get an accurate preview of changes with the current Kubernetes API is to allow patch permission. We will add documentation about this requirement to the provider, but I don’t see a path to avoiding it at this time.