Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[aws-ecs] TaskDefinition that uses cross account ECR repository image throws an error

See original GitHub issue

On v1.60.0(current latest release v.1.62.0 too), a TaskDefinition that uses cross account ECR repository image throws an error. On earlier v1.59.0, same code works fine.

Reproduction Steps

Use CDK v1.60.0

import * as cdk from "@aws-cdk/core";
import * as ecr from "@aws-cdk/aws-ecr";
import * as ecs from "@aws-cdk/aws-ecs";

const app = new cdk.App();
const region = "us-east-1";
const stackA = new cdk.Stack(app, "stackA", {
  env: {
    region,
    account: "111111111111",
  },
});
const ecrRepo = new ecr.Repository(stackA, "testEcrRepo", {
  repositoryName: "test-ecr-repo",
});

const stackB = new cdk.Stack(app, "stackB", {
  env: {
    region,
    account: "222222222222",
  },
});
const taskDef = new ecs.FargateTaskDefinition(stackB, "testTaskDef");
taskDef.addContainer("test-container", {
  image: ecs.ContainerImage.fromEcrRepository(ecrRepo),
});

$ cdk --version
1.60.0 (build 8e3f53a)

$ cdk synth

/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/resource.ts:154
      throw new Error(`Cannot use resource '${this.node.path}' in a cross-environment fashion, ` +
            ^
Error: Resolution error: Resolution error: Resolution error: Resolution error: Resolution error: Cannot use resource 'stackB/testTaskDef/ExecutionRole' in a cross-environment fashion, the resource's physical name must be explicit set or use `PhysicalName.GENERATE_IF_NEEDED`..

What did you expect to happen?

cdk synth should succeeds like earlier v1.59.0

$ cdk --version
1.59.0 (build 1d082f4)

$ cdk synth stackA
Resources:
  testEcrRepo66D8A784:
    Type: AWS::ECR::Repository
    Properties:
      RepositoryName: test-ecr-repo
    UpdateReplacePolicy: Retain
    DeletionPolicy: Retain
    Metadata:
      aws:cdk:path: stackA/testEcrRepo/Resource

$ cdk synth stackB
Resources:
  testTaskDefTaskRole8C17AF3A:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: ecs-tasks.amazonaws.com
        Version: "2012-10-17"
    Metadata:
      aws:cdk:path: stackB/testTaskDef/TaskRole/Resource
  testTaskDef147F11A0:
    Type: AWS::ECS::TaskDefinition
    Properties:
      ContainerDefinitions:
        - Essential: true
          Image:
            Fn::Join:
              - ""
              - - Fn::Select:
                    - 4
                    - Fn::Split:
                        - ":"
                        - Fn::Join:
                            - ""
                            - - "arn:"
                              - Ref: AWS::Partition
                              - :ecr:us-east-1:111111111111:repository/test-ecr-repo
                - .dkr.ecr.
                - Fn::Select:
                    - 3
                    - Fn::Split:
                        - ":"
                        - Fn::Join:
                            - ""
                            - - "arn:"
                              - Ref: AWS::Partition
                              - :ecr:us-east-1:111111111111:repository/test-ecr-repo
                - "."
                - Ref: AWS::URLSuffix
                - /test-ecr-repo:latest
          Name: test-container
      Cpu: "256"
      ExecutionRoleArn:
        Fn::GetAtt:
          - testTaskDefExecutionRole814B3A38
          - Arn
      Family: stackBtestTaskDef150CF57D
      Memory: "512"
      NetworkMode: awsvpc
      RequiresCompatibilities:
        - FARGATE
      TaskRoleArn:
        Fn::GetAtt:
          - testTaskDefTaskRole8C17AF3A
          - Arn
    Metadata:
      aws:cdk:path: stackB/testTaskDef/Resource
  testTaskDefExecutionRole814B3A38:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: ecs-tasks.amazonaws.com
        Version: "2012-10-17"
    Metadata:
      aws:cdk:path: stackB/testTaskDef/ExecutionRole/Resource
  testTaskDefExecutionRoleDefaultPolicy270D14AB:
    Type: AWS::IAM::Policy
    Properties:
      PolicyDocument:
        Statement:
          - Action:
              - ecr:BatchCheckLayerAvailability
              - ecr:GetDownloadUrlForLayer
              - ecr:BatchGetImage
            Effect: Allow
            Resource:
              Fn::Join:
                - ""
                - - "arn:"
                  - Ref: AWS::Partition
                  - :ecr:us-east-1:111111111111:repository/test-ecr-repo
          - Action: ecr:GetAuthorizationToken
            Effect: Allow
            Resource: "*"
        Version: "2012-10-17"
      PolicyName: testTaskDefExecutionRoleDefaultPolicy270D14AB
      Roles:
        - Ref: testTaskDefExecutionRole814B3A38
    Metadata:
      aws:cdk:path: stackB/testTaskDef/ExecutionRole/DefaultPolicy/Resource

What actually happened?

cdk synth fails with an error.

Full stack trace 
/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/resource.ts:154
      throw new Error(`Cannot use resource '${this.node.path}' in a cross-environment fashion, ` +
            ^
Error: Resolution error: Resolution error: Resolution error: Resolution error: Resolution error: Cannot use resource 'stackB/testTaskDef/ExecutionRole' in a cross-environment fashion, the resource's physical name must be explicit set or use `PhysicalName.GENERATE_IF_NEEDED`..

Object creation stack:
  at new PolicyDocument (/workspaces/aws-infra/node_modules/@aws-cdk/aws-iam/lib/policy-document.ts:48:30)
  at Repository.addToResourcePolicy (/workspaces/aws-infra/node_modules/@aws-cdk/aws-ecr/lib/repository.ts:471:29)
  at Function.addToPrincipalOrResource (/workspaces/aws-infra/node_modules/@aws-cdk/aws-iam/lib/grant.ts:147:45)
  at Repository.grant (/workspaces/aws-infra/node_modules/@aws-cdk/aws-ecr/lib/repository.ts:228:22)
  at Repository.grantPull (/workspaces/aws-infra/node_modules/@aws-cdk/aws-ecr/lib/repository.ts:241:22)
  at EcrImage.bind (/workspaces/aws-infra/node_modules/@aws-cdk/aws-ecs/lib/images/ecr.ts:29:21)
  at new ContainerDefinition (/workspaces/aws-infra/node_modules/@aws-cdk/aws-ecs/lib/container-definition.ts:382:36)
  at FargateTaskDefinition.addContainer (/workspaces/aws-infra/node_modules/@aws-cdk/aws-ecs/lib/base/task-definition.ts:387:12)
  at Object.<anonymous> (/workspaces/aws-infra/src/app2.ts:24:9)
  at Module._compile (internal/modules/cjs/loader.js:1137:30)
  at Module.m._compile (/workspaces/aws-infra/node_modules/ts-node/src/index.ts:858:23)
  at Module._extensions..js (internal/modules/cjs/loader.js:1157:10)
  at Object.require.extensions.<computed> [as .ts] (/workspaces/aws-infra/node_modules/ts-node/src/index.ts:861:12)
  at Module.load (internal/modules/cjs/loader.js:985:32)
  at Function.Module._load (internal/modules/cjs/loader.js:878:14)
  at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
  at main (/workspaces/aws-infra/node_modules/ts-node/src/bin.ts:227:14)
  at Object.<anonymous> (/workspaces/aws-infra/node_modules/ts-node/src/bin.ts:513:3)
  at Module._compile (internal/modules/cjs/loader.js:1137:30)
  at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10)
  at Module.load (internal/modules/cjs/loader.js:985:32)
  at Function.Module._load (internal/modules/cjs/loader.js:878:14)
  at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
  at internal/main/run_main_module.js:17:47.
Object creation stack:
  at new LazyBase (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/lazy.ts:126:26)
  at new LazyAny (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/lazy.ts:181:5)
  at Function.anyValue (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/lazy.ts:115:12)
  at new Repository (/workspaces/aws-infra/node_modules/@aws-cdk/aws-ecr/lib/repository.ts:420:34)
  at Object.<anonymous> (/workspaces/aws-infra/src/app2.ts:13:17)
  at Module._compile (internal/modules/cjs/loader.js:1137:30)
  at Module.m._compile (/workspaces/aws-infra/node_modules/ts-node/src/index.ts:858:23)
  at Module._extensions..js (internal/modules/cjs/loader.js:1157:10)
  at Object.require.extensions.<computed> [as .ts] (/workspaces/aws-infra/node_modules/ts-node/src/index.ts:861:12)
  at Module.load (internal/modules/cjs/loader.js:985:32)
  at Function.Module._load (internal/modules/cjs/loader.js:878:14)
  at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
  at main (/workspaces/aws-infra/node_modules/ts-node/src/bin.ts:227:14)
  at Object.<anonymous> (/workspaces/aws-infra/node_modules/ts-node/src/bin.ts:513:3)
  at Module._compile (internal/modules/cjs/loader.js:1137:30)
  at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10)
  at Module.load (internal/modules/cjs/loader.js:985:32)
  at Function.Module._load (internal/modules/cjs/loader.js:878:14)
  at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
  at internal/main/run_main_module.js:17:47.
Object creation stack:
  at new Intrinsic (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/private/intrinsic.ts:28:26)
  at new PostResolveToken (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/util.ts:83:5)
  at Object.ignoreEmpty (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/util.ts:38:10)
  at CfnRepository._toCloudFormation (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/cfn-resource.ts:297:25)
  at /workspaces/aws-infra/node_modules/@aws-cdk/core/lib/private/refs.ts:120:58
  at Object.findTokens (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/private/resolve.ts:174:11)
  at findAllReferences (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/private/refs.ts:120:22)
  at Object.resolveReferences (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/private/refs.ts:24:17)
  at Object.prepareApp (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/private/prepare-app.ts:36:5)
  at Object.synthesize (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/private/synthesis.ts:21:3)
  at App.synth (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/stage.ts:175:23)
  at process.<anonymous> (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/app.ts:112:45)
  at Object.onceWrapper (events.js:422:26)
  at process.emit (events.js:315:20)
  at process.EventEmitter.emit (domain.js:483:12)
  at process.emit (/workspaces/aws-infra/node_modules/source-map-support/source-map-support.js:495:21).
Object creation stack:
  at new Intrinsic (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/private/intrinsic.ts:28:26)
  at new PostResolveToken (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/util.ts:83:5)
  at CfnRepository._toCloudFormation (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/cfn-resource.ts:295:29)
  at /workspaces/aws-infra/node_modules/@aws-cdk/core/lib/private/refs.ts:120:58
  at Object.findTokens (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/private/resolve.ts:174:11)
  at findAllReferences (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/private/refs.ts:120:22)
  at Object.resolveReferences (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/private/refs.ts:24:17)
  at Object.prepareApp (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/private/prepare-app.ts:36:5)
  at Object.synthesize (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/private/synthesis.ts:21:3)
  at App.synth (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/stage.ts:175:23)
  at process.<anonymous> (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/app.ts:112:45)
  at Object.onceWrapper (events.js:422:26)
  at process.emit (events.js:315:20)
  at process.EventEmitter.emit (domain.js:483:12)
  at process.emit (/workspaces/aws-infra/node_modules/source-map-support/source-map-support.js:495:21)
    at Role._enableCrossEnvironment (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/resource.ts:154:13)
    at Object.resolve (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/resource.ts:217:16)
    at RememberingTokenResolver.resolveToken (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/resolvable.ts:134:24)
    at RememberingTokenResolver.resolveToken (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/private/resolve.ts:187:18)
    at resolve (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/private/resolve.ts:133:29)
    at Object.resolve [as mapToken] (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/private/resolve.ts:49:32)
    at TokenizedStringFragments.mapTokens (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/string-fragments.ts:71:33)
    at RememberingTokenResolver.resolveString (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/resolvable.ts:155:22)
    at RememberingTokenResolver.resolveString (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/private/resolve.ts:191:23)
    at resolve (/workspaces/aws-infra/node_modules/@aws-cdk/core/lib/private/resolve.ts:91:31)

Environment

  • CLI Version :1.60.0
  • Framework Version:1.60.0
  • Node.js Version:v12.18.3
  • OS :Linux(Docker)
  • Language (Version):TypeScript (3.8.3)

Other

I suspect changes from https://github.com/aws/aws-cdk/pull/8280 affected this issue.

This is 🐛 Bug Report

Issue Analytics

  • State:open
  • Created 3 years ago
  • Reactions:9
  • Comments:5 (2 by maintainers)

github_iconTop GitHub Comments

5reactions
garyd203commented, Mar 3, 2021

For people who are want to solve this bug for cross-account ECR usage (as per original bug report), I can’t find a way to set the name of the internally generated execution role for the task definition. However, if you create an empty role then the TaskDefinition construct will add the necessary policies. Example in Python:

role = Role(
    scope,
    "ExecRole",
    assumed_by=ServicePrincipal(service="ecs-tasks.amazonaws.com"),
    role_name=PhysicalName.GENERATE_IF_NEEDED,
)

task_definition = FargateTaskDefinition(
    scope,
    "TaskDef",
    execution_role=role,
    family="my-task-family"
)

However, note that CDK will create a ECR repository policy that explicitly references the cross-account execution role, and ECR will validate this when you set the policy. This creates a circular deployment dependency where the IAM role needs to be deployed before the ECR repository, but the IAM role is deployed with an ECS task definition that depends on the ECR repository.

I believe you can break the circularity by doing an initial deployment with the (unused) IAM role, and then a second full deployment. Alternatively, I broke the circularity by forcing CDK to discard the over-constrained repository policy and setting my own using just the AWS account, like so:

ecr_repo.node.default_child.add_property_override(
        "RepositoryPolicyText",
        PolicyDocument(
            statements=[
                PolicyStatement(
                    actions=[
                        "ecr:BatchCheckLayerAvailability",
                        "ecr:BatchGetImage",
                        "ecr:GetDownloadUrlForLayer",
                    ],
                    effect=Effect.ALLOW,
                    principals=[AccountPrincipal(other_account_number)],
                )
            ],
        ).to_json(),
    )

In terms of security impact, for standard multiple-account use cases, It’s hard to argue that there’s any benefit to a policy that allows reads from only some roles in another account - the implication is that there are some roles in the other account which should be explicitly prohibited from accessing the repository and are also incapable of gaining access to one of the permitted roles, which suggests that the other account has multiple incompatible purposes. OTOH having a decoupled policy is a significant benefit for maintainability and deployability.

1reaction
iamhopaul123commented, Jan 5, 2021

Hello @idm-ryou, @abbottdev, seems like there is already a pretty good explanation for the reason why this error occurs and solution to how to reference a cross account/region resource: https://stackoverflow.com/questions/62989129/resolution-error-cannot-use-resource-x-in-a-cross-environment-fashion-the-re.

I would recommend using the same account/region. However, if you want to reuse the repository, you could deploy stackA and get the ARN for the ECR repo then hardcode the repo ARN in stackB task definition. Please let me know if this still won’t work for you.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Resolve the "Image does not exist" error in an Amazon ECS ...
The resolution for this error applies to pulling images from an Amazon Elastic Container Registry (Amazon ECR) repository.
Read more >
Resolve the Amazon ECR error "CannotPullContainerError
If I pull images with Amazon Elastic Container Registry (Amazon ECR), I get the following error: "CannotPullContainerError: API error." How can ...
Read more >
Troubleshooting Amazon ECR error messages
If you specify a Docker Hub repository that does not currently exist, Docker Hub creates it automatically. With Amazon ECR, new repositories must...
Read more >
Allow secondary accounts to push or pull images in Amazon ...
You can push or pull images to or from an Amazon ECR repository in ... task definition, set the image that you want...
Read more >
Allow Amazon ECS tasks to pull images from an Amazon ECR ...
To access the Amazon ECR image repository with your launch type, choose one of these options: For Amazon Elastic Compute Cloud (Amazon EC2) ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

[aws-eks] Install helm chart from asset

Next page

[aws-apigatewayv2] HttpMethod.ANY throws error in deployment when specifying AllowedMethods of CORS configuration