TarballImageAsset `executable` command in `...assets.json` prevents running `cdk deploy` outside cloud assembly root directory

We were very excited to use the new TarballImageAsset class to deploy docker images built by Kaniko in our CI pipeline, but it seems to break deployments unless run in a specific and inconvenient way.

Reproduction Steps

Given the following CDK project, where hello-world.tar is produced with docker save hello-world -o hello-world.tar:

.
├── app.py
├── cdk.json
└── hello-world.tar

app.py:

from aws_cdk import core as cdk
from aws_cdk.aws_ecs import ContainerImage, FargateTaskDefinition

app = cdk.App()

stack = cdk.Stack(app, "TestStack")

task_def = FargateTaskDefinition(
    stack, "TestTaskDef",
)
task_def.add_container(
    "TestContainer",
    image=ContainerImage.from_tarball("hello-world.tar"),
)

app.synth()

cdk.json:

{
  "app": "python3 app.py",
  "context": {
    "@aws-cdk/aws-ecr-assets:dockerIgnoreSupport": true,
    "@aws-cdk/aws-ecs-patterns:removeDefaultDesiredCount": true,
    "@aws-cdk/aws-kms:defaultKeyPolicies": true,
    "@aws-cdk/aws-s3:grantWriteWithoutAcl": true,
    "@aws-cdk/aws-secretsmanager:parseOwnedSecretName": true,
    "@aws-cdk/core:enableStackNameDuplicates": "true",
    "@aws-cdk/core:stackRelativeExports": "true",
    "@aws-cdk/core:newStyleStackSynthesis": true,
    "aws-cdk:enableDiffNoFail": "true"
  }
}

Then, run cdk depoy -vvv from the project root.

What did you expect to happen?

The deployment should succeed, creating an ECS task definition from the docker image tarball.

What actually happened?

Deployment fails with the verbose output shown below.

The root cause appears to be that the executable command (in cdk.out/TestStack.assets.json, shown below) is run from the same working directory as the cdk deploy command - obviously the asset...tar file is not present when run from the project root.

...
      "source": {
        "executable": [
          "sh",
          "-c",
          "docker load -i asset.c2a72eb491044b4b079b0430e01e40d428a629b736f0e3c61f2f51170ed20069.tar | sed \"s/Loaded image: //g\""
        ]
      },
...

This seems to be confirmed by the fact that when I first run cdk synth, and then cd cdk.out; cdk deploy --app ., the deployment succeeds.

Suggestions

It seems very strange to me that the deployment operation is so dependent on the current working directory - I would expect always to be able to run cdk deploy --app /path/to/cloud/assembly and have it work.

Ideally, asset executable commands would be run from the cloud assembly directory, regardless of where the cdk deploy command was run from. If changing this would break backwards compatibility, then a possible solution might be to provide the path to the cloud assembly root to the executable somehow, e.g. as an environment variable. The executable written by TarballImageAsset could then be something like:

"executable": ["sh", "-c", "docker load -i $CDK_CLOUD_ASM_ROOT/asset.c2a...069.tar | sed \"s/Loaded image: //g\""]

In any case there are a couple of other issues I can see that made it very hard to understand what is going wrong here:

• Despite the docker load command failing, the exit code of the command overall is 0 because of the pipe to sed, causing execution to continue past where it should.
• Similarly, the output from the command is the empty string, which the CLI then tries to pass as an argument to docker tag, causing the error seen below. Ideally the output of the executable should be validated as soon as it returns.

Thanks for reading, and if you would be willing to accept a PR along the lines of any of these suggestions then let me know.

Environment

• CDK CLI Version : 1.114.0
• Framework Version: 1.114.0
• Node.js Version: 15.14.0
• OS : Linux
• Language (Version): Python (3.8.5)

Other

Verbose cdk deploy output:

...
TestStack: deploying...
Retrieved account ID XXXXXXXXXXXX from disk cache
Assuming role 'arn:aws:iam::XXXXXXXXXXXX:role/cdk-hnb659fds-deploy-role-XXXXXXXXXXXX-eu-west-2'.
Waiting for stack CDKToolkit to finish creating or updating...
[AWS cloudformation 200 0.142s 0 retries] describeStacks({ StackName: 'CDKToolkit' })
[AWS ssm 200 0.207s 0 retries] getParameter({ Name: '/cdk-bootstrap/hnb659fds/version' })
[0%] start: Publishing b04e11e72beb3cf24c3704c221e4f85f380262ec35ef875764f306f4eff0e644:current_account-current_region
Retrieved account ID XXXXXXXXXXXX from disk cache
Retrieved account ID XXXXXXXXXXXX from disk cache
Assuming role 'arn:aws:iam::XXXXXXXXXXXX:role/cdk-hnb659fds-file-publishing-role-XXXXXXXXXXXX-eu-west-2'.
[0%] check: Check s3://cdk-hnb659fds-assets-XXXXXXXXXXXX-eu-west-2/b04e11e72beb3cf24c3704c221e4f85f380262ec35ef875764f306f4eff0e644.json
[AWS s3 200 0.194s 0 retries] getBucketLocation({ Bucket: 'cdk-hnb659fds-assets-XXXXXXXXXXXX-eu-west-2' })
[AWS s3 200 0.036s 0 retries] listObjectsV2({
  Bucket: 'cdk-hnb659fds-assets-XXXXXXXXXXXX-eu-west-2',
  Prefix: 'b04e11e72beb3cf24c3704c221e4f85f380262ec35ef875764f306f4eff0e644.json',
  MaxKeys: 1
})
[0%] found: Found s3://cdk-hnb659fds-assets-XXXXXXXXXXXX-eu-west-2/b04e11e72beb3cf24c3704c221e4f85f380262ec35ef875764f306f4eff0e644.json
[50%] success: Published b04e11e72beb3cf24c3704c221e4f85f380262ec35ef875764f306f4eff0e644:current_account-current_region
[50%] start: Publishing c2a72eb491044b4b079b0430e01e40d428a629b736f0e3c61f2f51170ed20069:current_account-current_region
Retrieved account ID XXXXXXXXXXXX from disk cache
Retrieved account ID XXXXXXXXXXXX from disk cache
Assuming role 'arn:aws:iam::XXXXXXXXXXXX:role/cdk-hnb659fds-image-publishing-role-XXXXXXXXXXXX-eu-west-2'.
[AWS ecr 200 0.163s 0 retries] describeRepositories({
  repositoryNames: [
    'cdk-hnb659fds-container-assets-XXXXXXXXXXXX-eu-west-2',
    [length]: 1
  ]
})
[50%] check: Check XXXXXXXXXXXX.dkr.ecr.eu-west-2.amazonaws.com/cdk-hnb659fds-container-assets-XXXXXXXXXXXX-eu-west-2:c2a72eb491044b4b079b0430e01e40d428a629b736f0e3c61f2f51170ed20069
[AWS ecr 400 0.025s 0 retries] describeImages({
  repositoryName: 'cdk-hnb659fds-container-assets-XXXXXXXXXXXX-eu-west-2',
  imageIds: [
    {
      imageTag: 'c2a72eb491044b4b079b0430e01e40d428a629b736f0e3c61f2f51170ed20069'
    },
    [length]: 1
  ]
})
Call failed: describeImages({"repositoryName":"cdk-hnb659fds-container-assets-XXXXXXXXXXXX-eu-west-2","imageIds":[{"imageTag":"c2a72eb491044b4b079b0430e01e40d428a629b736f0e3c61f2f51170ed20069"}]}) => The image with imageId {imageDigest:'null', imageTag:'c2a72eb491044b4b079b0430e01e40d428a629b736f0e3c61f2f51170ed20069'} does not exist within the repository with name 'cdk-hnb659fds-container-assets-XXXXXXXXXXXX-eu-west-2' in the registry with id 'XXXXXXXXXXXX' (code=ImageNotFoundException)
[AWS ecr 200 0.015s 0 retries] getAuthorizationToken({})
[50%] debug: docker login --username AWS --password-stdin https://XXXXXXXXXXXX.dkr.ecr.eu-west-2.amazonaws.com
[50%] build: Building Docker image using command 'sh,-c,docker load -i asset.c2a72eb491044b4b079b0430e01e40d428a629b736f0e3c61f2f51170ed20069.tar | sed "s/Loaded image: //g"'
[50%] upload: Push XXXXXXXXXXXX.dkr.ecr.eu-west-2.amazonaws.com/cdk-hnb659fds-container-assets-XXXXXXXXXXXX-eu-west-2:c2a72eb491044b4b079b0430e01e40d428a629b736f0e3c61f2f51170ed20069
[50%] debug: docker tag  XXXXXXXXXXXX.dkr.ecr.eu-west-2.amazonaws.com/cdk-hnb659fds-container-assets-XXXXXXXXXXXX-eu-west-2:c2a72eb491044b4b079b0430e01e40d428a629b736f0e3c61f2f51170ed20069
Error parsing reference: "" is not a valid repository/tag: invalid reference format
[100%] fail: docker tag  XXXXXXXXXXXX.dkr.ecr.eu-west-2.amazonaws.com/cdk-hnb659fds-container-assets-XXXXXXXXXXXX-eu-west-2:c2a72eb491044b4b079b0430e01e40d428a629b736f0e3c61f2f51170ed20069 exited with error code 1: Error parsing reference: "" is not a valid repository/tag: invalid reference format

 ❌  TestStack failed: Error: Failed to publish one or more assets. See the error messages above for more information.
    at Object.publishAssets (/home/name/.nvm/versions/node/v15.14.0/lib/node_modules/aws-cdk/lib/util/asset-publishing.ts:25:11)
    at processTicksAndRejections (node:internal/process/task_queues:94:5)
    at CloudFormationDeployments.publishStackAssets (/home/name/.nvm/versions/node/v15.14.0/lib/node_modules/aws-cdk/lib/api/cloudformation-deployments.ts:278:7)
    at CloudFormationDeployments.deployStack (/home/name/.nvm/versions/node/v15.14.0/lib/node_modules/aws-cdk/lib/api/cloudformation-deployments.ts:179:5)
    at CdkToolkit.deploy (/home/name/.nvm/versions/node/v15.14.0/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:184:24)
    at initCommandLine (/home/name/.nvm/versions/node/v15.14.0/lib/node_modules/aws-cdk/bin/cdk.ts:213:9)
Failed to publish one or more assets. See the error messages above for more information.

---

This is 🐛 Bug Report

[eladb]

@njlynch @rix0rrr @otaviomacedo it seems like buildExternalAsset() runs the executable without setting the cwd. Would it make sense that cwd will be set to the cloud assembly root?

https://github.com/aws/aws-cdk/blob/a8b1c471b7058bbf739a1d4f5b4860656ebd5432/packages/cdk-assets/lib/private/handlers/container-images.ts#L86

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.