Automation API seemingly removes AWS policy attachments that do not belong to stack.

Expected behavior

In our project set up we have a main Pulumi project, as well as a “nested” Pulumi project and “child” stack that we use with the Automation API and Jest for spinning up temporary resources needed for running integration tests on the “parent” stack.

We expect that pulumi up for our main project will create those stack resources belonging to our main project, and that running our Automation API scripts will deploy and teardown the child stack and only its resources.

Current behavior

What we see happen is that after the Automation API scripts run, role policy attachments associated with lambdas in the parent stack are removed. Pulumi still considers those policy attachments as existing stack resources until running pulumi refresh on the parent stack.

Steps to reproduce

I have created a simple example repo for recreating this behavior. https://github.com/andrewlaskey/pulumi-example

There is a main project that can be deployed with pulumi up that creates a lambda with two policies attached. Then there is a script that can be run, which uses the Automation API to create another stack with a lambda, policies, and a stack reference to the “parent” stack.

1. Set AWS profile
2. Deploy main project stack: pulumi stack select parent-stack and pulumi up
3. Confirm deployment by looking in Pulumi dashboard and role in AWS console.
4. Run the automation API: npm run automation
5. Run the teardown script: npm run automation:destroy
6. Refreshing the page in the AWS console for the role will show that the policies are gone.
7. pulumi up in the main project will not recreate those policies and only after pulumi refresh and pulumi up will the main stack be corrected again.

Example

     Type                         Name                                                            Status      
 +   pulumi:pulumi:Stack          disappearing-policies-parent-stack                              created     
 +   ├─ aws:iam:Role              pulumiPolicyTest-role                                           created     
 +   ├─ aws:iam:PolicyAttachment  pulumiPolicyTest-lambda-AWSLambdaBasicExecutionRole-policy      created     
 +   ├─ aws:iam:PolicyAttachment  pulumiPolicyTest-lambda-AWSLambdaVPCAccessExecutionRole-policy  created     
 +   └─ aws:lambda:Function       pulumiPolicyTest                                                created     
 
Outputs:
    testLambdaName: "pulumiPolicyTest-aabdd56"
    testLambdaRole: "pulumiPolicyTest-role-71b7bf4"

Resources:
    + 5 created

Duration: 21s

> automation:destroy
> ts-node scripts.ts --destroy

******** CHILD INFRA TEARDOWN ********
Destroying stack: child-stack
Destroying (child-stack)


View Live: https://app.pulumi.com/andrewlaskey/disappearing-policies-child-infra/child-stack/updates/3




 -  aws:iam:PolicyAttachment pulumiPolicyTestChild-lambda-AWSLambdaBasicExecutionRole-policy deleting 
 -  aws:iam:PolicyAttachment pulumiPolicyTestChild-lambda-AWSLambdaVPCAccessExecutionRole-policy deleting 

 -  aws:lambda:Function pulumiPolicyTestChild deleting 

 -  aws:lambda:Function pulumiPolicyTestChild deleted 

 -  aws:iam:PolicyAttachment pulumiPolicyTestChild-lambda-AWSLambdaBasicExecutionRole-policy deleted 

 -  aws:iam:PolicyAttachment pulumiPolicyTestChild-lambda-AWSLambdaVPCAccessExecutionRole-policy deleted 

 -  aws:iam:Role pulumiPolicyTestChild-role deleting 

 -  aws:iam:Role pulumiPolicyTestChild-role deleted 

 -  pulumi:pulumi:Stack disappearing-policies-child-infra-child-stack deleting 

 -  pulumi:pulumi:Stack disappearing-policies-child-infra-child-stack deleted 
 

Outputs:
  - integrationLambdaName: "pulumiPolicyTestChild-f5e373d"

Resources:
    - 5 deleted

Duration: 8s

Context (Environment)

This is affecting the development of our project and our ability to create temporary resources for the purpose of integration testing our main project. We could reconfigure the type of tests we are running so that we don’t need ephemeral infrastructure, however, that would limit the kind of tests we can run. We would also like to better understand the ways in which nested Pulumi projects can possibly interact with each other so we can avoid this pitfall if we need to follow this pattern for some other reason.

Thanks!

[leezen]

This is very odd. Thank you for the detailed repro. When I look at the child stack checkpoint, I see the following:

{
                "urn": "urn:pulumi:child-stack::disappearing-policies-child-infra::aws:iam/policyAttachment:PolicyAttachment::pulumiPolicyTestChild-lambda-AWSLambdaBasicExecutionRole-policy",
                "custom": true,
                "id": "pulumiPolicyTestChild-lambda-AWSLambdaBasicExecutionRole-policy-72a42b1",
                "type": "aws:iam/policyAttachment:PolicyAttachment",
                "inputs": {
                    "__defaults": ["name"],
                    "name": "pulumiPolicyTestChild-lambda-AWSLambdaBasicExecutionRole-policy-72a42b1",
                    "policyArn": "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole",
                    "roles": ["pulumiPolicyTestChild-role-1015108"]
                },
                "outputs": {
                    "groups": [],
                    "id": "pulumiPolicyTestChild-lambda-AWSLambdaBasicExecutionRole-policy-72a42b1",
                    "name": "pulumiPolicyTestChild-lambda-AWSLambdaBasicExecutionRole-policy-72a42b1",
                    "policyArn": "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole",
                    "roles": [
                        "pulumiPolicyTest-role-dec4db6",
                        "pulumiPolicyTestChild-role-1015108"
                    ],
                    "users": []
                },
                "parent": "urn:pulumi:child-stack::disappearing-policies-child-infra::pulumi:pulumi:Stack::disappearing-policies-child-infra-child-stack",
                "dependencies": [
                    "urn:pulumi:child-stack::disappearing-policies-child-infra::aws:iam/role:Role::pulumiPolicyTestChild-role"
                ],
                "provider": "urn:pulumi:child-stack::disappearing-policies-child-infra::pulumi:providers:aws::default_3_30_1::594f85bb-3557-4bde-a832-8536c9986d25",
                "propertyDependencies": {
                    "policyArn": null,
                    "roles": [
                        "urn:pulumi:child-stack::disappearing-policies-child-infra::aws:iam/role:Role::pulumiPolicyTestChild-role"
                    ]
                }
            },

So, when we destroy the child stack, I’m guessing what’s happening is it’s removing the association with both roles. The verify this hypothesis, I removed the reference to pulumiPolicyTest-role-dec4db6 and imported that checkpoint. Then, I ran npm run automation:destroy and verified that indeed the original pulumiPolicyTest-role-dec4db6 still has the policies attached on the cloud provider side. Now, the question is, why does pulumiPolicyTest-role-dec4db6 get referenced in the child stack for the policy association? Need to dig further into that, but thought I’d at least drop a note indicating what I’ve found so far.

[andrewlaskey]

Actually, I’m not sure why I didn’t notice this at the outset, but the problem here is PolicyAttachment – this is applied to a policy which is the same policy in the case of the child and parent programs. You’re modifying what roles a particular policy is attached to. Instead, what you actually want to use is RolePolicyAttachment which applies a policy to a role.

Wow, yep that was the problem. 🤦

Thanks for finding that!