[aws-eks] ServiceAccount property references do not result in dependencies

CDK does not currently establish any sort of dependency–implicit or otherwise–between a ServiceAccount construct and any EKS resources that refer to its properties, such as HelmCharts and KubernetesManifests. This can result in the service account being destroyed before a dependent CRD controller is destroyed. This can cause ordering failures on stack destruction.

Consider the following sequence of events:

1. ServiceAccount is destroyed
2. Controller (which uses the service account) is still running but all operations will fail
3. CRD (which depends on controller’s proper operation) cannot be successfully deleted

At root, the issue seems to be that ServiceAccounts are not associated with first-class Custom Resources. The name and namespace parameters are not formal CloudFormation parameters, which can be used as references by other CF Resources. Instead, an ordinary KubernetesResource is created, and the name and namespace parameters are interned into the K8S resource JSON.

Nor does CDK create any explicit dependencies between a service account and its consumer by way of a DependsOn key in the rendered template.

❗ In fact, further testing shows that it is not even possible to explicitly declare a dependency between a ServiceAccount and a dependent resource. No error results, but no dependency is created, either.

Reproduction Steps

      const appMeshControllerServiceAccount = cluster.addServiceAccount(
        "AppMesh",
        {
          name: "appmesh-controller",
          namespace: "appmesh-system",
        }
      );

      const appMeshController = cluster.addChart("appmesh-controller", {
        chart: "appmesh-controller",
        repository: "https://aws.github.io/eks-charts",
        namespace: "appmesh-system",
        createNamespace: false,
        release: "appmesh-controller",
        values: {
          region: this.region,
          serviceAccount: {
            create: false,
            // Expectation is that the following creates an implicit dependency
            name: appMeshControllerServiceAccount.serviceAccountName,
          },
        },
        wait: true,
      });

Rendered template fragment:

"ClusterAmanifestAppMeshServiceAccountResourceAABB6973": {
      "Type": "Custom::AWSCDK-EKS-KubernetesResource",
      "Properties": {
        "ServiceToken": {
          "Fn::GetAtt": [
            "awscdkawseksKubectlProviderNestedStackawscdkawseksKubectlProviderNestedStackResourceA7AEBA6B",
            "Outputs.LocalZoneAppMeshRoutingStackawscdkawseksKubectlProviderframeworkonEvent9E9D6AA9Arn"
          ]
        },
        "Manifest": {
          "Fn::Join": [
            "",
            [
              "[{\"apiVersion\":\"v1\",\"kind\":\"ServiceAccount\",\"metadata\":{\"name\":\"appmesh-controller\",\"namespace\":\"appmesh-system\",\"labels\":{\"app.kubernetes.io/name\":\"appmesh-controller\"},\"annotations\":{\"eks.amazonaws.com/role-arn\":\"",
              {
                "Fn::GetAtt": [
                  "ClusterAAppMeshRole998D0403",
                  "Arn"
                ]
              },
              "\"}}}]"
            ]
          ]
        },
        "ClusterName": {
          "Ref": "ClusterA66FB302D"
        },
        "RoleArn": {
          "Fn::GetAtt": [
            "ClusterACreationRoleDCD3F752",
            "Arn"
          ]
        }
      },
      "DependsOn": [
        "ClusterAKubectlReadyBarrier407D31C8"
      ],
      "UpdateReplacePolicy": "Delete",
      "DeletionPolicy": "Delete",
      "Metadata": {
        "aws:cdk:path": "LocalZoneAppMeshRoutingStack/ClusterA/manifest-AppMeshServiceAccountResource/Resource/Default"
      }
    },
    "ClusterAchartappmeshcontrollerBC62F8B6": {
      "Type": "Custom::AWSCDK-EKS-HelmChart",
      "Properties": {
        "ServiceToken": {
          "Fn::GetAtt": [
            "awscdkawseksKubectlProviderNestedStackawscdkawseksKubectlProviderNestedStackResourceA7AEBA6B",
            "Outputs.LocalZoneAppMeshRoutingStackawscdkawseksKubectlProviderframeworkonEvent9E9D6AA9Arn"
          ]
        },
        "ClusterName": {
          "Ref": "ClusterA66FB302D"
        },
        "RoleArn": {
          "Fn::GetAtt": [
            "ClusterACreationRoleDCD3F752",
            "Arn"
          ]
        },
        "Release": "appmesh-controller",
        "Chart": "appmesh-controller",
        "Wait": true,
        "Values": "{\"region\":\"us-east-1\",\"serviceAccount\":{\"create\":false,\"name\":\"appmesh-controller\"}}",
        "Namespace": "appmesh-system",
        "Repository": "https://aws.github.io/eks-charts"
      },
      "DependsOn": [
        "ClusterAKubectlReadyBarrier407D31C8",
        "ClusterAmanifestAppMeshSystemNamespace65398EC6",
        "ClusterANodegroupNodeGroupRoleBC94C691",
        "ClusterANodegroup47FA0B14"
      ],
      "UpdateReplacePolicy": "Delete",
      "DeletionPolicy": "Delete",
      "Metadata": {
        "aws:cdk:path": "LocalZoneAppMeshRoutingStack/ClusterA/chart-appmesh-controller/Resource/Default"
      }
    },

What did you expect to happen?

I expected an implicit dependency to be created on the serviceAccount by the chart.

What actually happened?

No such dependency was created.

Environment

• CLI Version : 1.59.0
• Framework Version: 1.59.0
• Node.js Version: 12.18.2
• OS : MacOS Catalina
• Language (Version): Typescript 3.7.5

Other

---

This is 🐛 Bug Report

[otterley]

One solution to this would be to actually have serviceAccount.serviceAccountName be a token that fetches the service account name by using the KubernetesObjectValue construct.

This would force the retrieval of .serviceAccountName to wait until the resource is actually created, but also feels like kind of a stretch since in many cases that won’t be necessary.

I disagree that it’s unnecessary. While it may be unnecessary for stack creation, it is quite often needed for stack destruction to work properly. Any K8S CRD controller that relies on a service account to work properly must not have its service account destroyed before the controller itself is. Otherwise it will be impossible to delete the CRDs that are managed by the controller.

[github-actions[bot]]

This issue has not received any attention in 1 year. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.