(cli): cross account deployments still don't work

Reproduction Steps

CI account: 0147xxxxxxxx Target account: 1454xxxxxxx

Target account is bootstrapped to trust CI account with --trust 0147xxxxxxxx . IAM user in CI account graned permission iam:AssumeRole to all roles created by bootstrap stack.

Create CDK app with environment in target account and an S3 asset:

...
new Stack(app, 'Stack', { env: {account: "1454xxxxxxx", region: "eu-west-2" }});
...

Do cdk deploy from CI account.

What did you expect to happen?

Successfull deploy

What actually happened?

If we try to deploy it with cdk deploy -v following happens:

OrganizationStack
OrganizationStack: deploying...
Looking up default account ID from STS
Default account ID: 0147xxxxxx
Assuming role 'arn:aws:iam::1454xxxxxx:role/cdk-hnb659fds-deploy-role-1454xxxxx-eu-west-2'.
Waiting for stack CDKToolkit to finish creating or updating...
[0%] start: Publishing 4a3609ad912843e581892f37ae9d6fb0fa1648b547693aaa562b0119452b8956:145469417702-eu-west-2
[50%] fail: Need to perform AWS calls for account 1454xxxxxxx, but the current credentials are for 0147xxxxxx
[50%] start: Publishing 97b2255ceb5577d4534d825ed8407ced52d0f8917ce15f7a76d5c47aba9bd603:145469417702-eu-west-2
[100%] fail: Need to perform AWS calls for account 1454xxxxxxxx, but the current credentials are for 0147xxxxxx

Environment

• CDK CLI Version : 1.78.-
• Framework Version:
• Node.js Version: 14
• OS : Linux
• Language (Version): Typescript

Other

Interestingly, cdk-assets uploads just fine. Exact commands when run from CI:

  - yarn add -D aws-cdk@1.78.0 
  - node_modules/.bin/cdk synth --verbose
  - node_modules/.bin/cdk-assets publish -p cdk.out/OrganizationStack.assets.json -v
  - node_modules/.bin/cdk deploy --require-approval=never --all --verbose

Produce following output (tail):

$ node_modules/.bin/cdk synth --verbose
Successfully synthesized to /data/build/X-WKy9gs/0/dir/devops/iac/organization/cdk.out
Supply a stack id (OrganizationStack, SingleSignOnStack) to display its template.

$ node_modules/.bin/cdk-assets publish -p cdk.out/OrganizationStack.assets.json -v
verbose: Loaded manifest from cdk.out/OrganizationStack.assets.json: 2 assets found
info   : [0%] start: Publishing 4a3609ad912843e581892f37ae9d6fb0fa1648b547693aaa562b0119452b8956:1454xxxxxxx-eu-west-2
verbose: Assume arn:aws:iam::1454xxxxxxx:role/cdk-hnb659fds-file-publishing-role-1454xxxxxxx-eu-west-2
verbose: [0%] check: Check s3://cdk-hnb659fds-assets-1454xxxxxxx-eu-west-2/4a3609ad912843e581892f37ae9d6fb0fa1648b547693aaa562b0119452b8956.zip
verbose: [0%] found: Found s3://cdk-hnb659fds-assets-1454xxxxxxx-eu-west-2/4a3609ad912843e581892f37ae9d6fb0fa1648b547693aaa562b0119452b8956.zip
info   : [50%] success: Published 4a3609ad912843e581892f37ae9d6fb0fa1648b547693aaa562b0119452b8956:1454xxxxxxx-eu-west-2
info   : [50%] start: Publishing 97b2255ceb5577d4534d825ed8407ced52d0f8917ce15f7a76d5c47aba9bd603:1454xxxxxxx-eu-west-2
verbose: Assume arn:aws:iam::1454xxxxxxx:role/cdk-hnb659fds-file-publishing-role-1454xxxxxxx-eu-west-2
verbose: [50%] check: Check s3://cdk-hnb659fds-assets-1454xxxxxxx-eu-west-2/97b2255ceb5577d4534d825ed8407ced52d0f8917ce15f7a76d5c47aba9bd603
verbose: [50%] upload: Upload s3://cdk-hnb659fds-assets-1454xxxxxxx-eu-west-2/97b2255ceb5577d4534d825ed8407ced52d0f8917ce15f7a76d5c47aba9bd603
info   : [100%] success: Published 97b2255ceb5577d4534d825ed8407ced52d0f8917ce15f7a76d5c47aba9bd603:1454xxxxxxx-eu-west-2


$ node_modules/.bin/cdk deploy --require-approval=never --all --verbose
...
OrganizationStack
OrganizationStack: deploying...
Looking up default account ID from STS
Default account ID: 0147xxxxxxx
Assuming role 'arn:aws:iam::1454xxxxxxx:role/cdk-hnb659fds-deploy-role-1454xxxxxxx-eu-west-2'.
Waiting for stack CDKToolkit to finish creating or updating...
[0%] start: Publishing 4a3609ad912843e581892f37ae9d6fb0fa1648b547693aaa562b0119452b8956:1454xxxxxxx-eu-west-2
[50%] fail: Need to perform AWS calls for account 1454xxxxxxx, but the current credentials are for 0147xxxxxxx
[50%] start: Publishing 97b2255ceb5577d4534d825ed8407ced52d0f8917ce15f7a76d5c47aba9bd603:1454xxxxxxx-eu-west-2
[100%] fail: Need to perform AWS calls for account 1454xxxxxxx, but the current credentials are for 0147xxxxxxx

Looks like fix done in https://github.com/aws/aws-cdk/pull/11966 is incomplete.

/cc @rix0rrr , @scarytom , @polothy

---

This is 🐛 Bug Report

[polothy]

Maybe no one will see this comment, but I was just able to deploy cross account with no plugins 🎉

Thanks @redbaron and @rix0rrr for fixing this!

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.