[core] unable to run CDK diff in upgrading from 1.51.0 to 1.62.0: : Assets must be defined indirectly within a "Stage" or an "App" scope

I just upgraded my cdk version from 1.51.0 to 1.62.0, and I updated the code so that npm build runs successfully for the project. I then tried to do a diff of my stacks with cdk diff <stackname> and it failed with this error:

unable to determine cloud assembly output directory. Assets must be defined indirectly within a "Stage" or an "App" scope
Subprocess exited with error 1

In googling that I came across https://github.com/aws/aws-cdk/issues/9546 and I followed the directions to make sure all my cdk packages were pinned at the exact same version. after doing that I still ran into the error. I do use custom lambda functions packaged like so:

 const statusLambda = new lambda.Function(this, 'StatusLambda', {
            code: lambda.Code.asset(path.join(__dirname, '../lambdas/codepipeline_status_lambda')),
            handler: 'index.handler',
            timeout: cdk.Duration.seconds(300),
            runtime: lambda.Runtime.NODEJS_10_X,
            environment: {
                ACCESS_TOKEN: cdk.SecretValue.secretsManager(props.gitHubSecretArn).toString(),
                DATADOG_API_KEY: cdk.SecretValue.secretsManager(props.dataDogSecretArn).toString(),
                ENV_TYPE: props.env_type,
                STAGE_TYPE: props.stage_type,
                APP_URL: props.url,
            },
        })

which worked just fine before the upgrade, but now it still is failing with the same error, that issue i linked above mentioned that using lambda assets could cause this. and I am unsure of how to proceed to fix this, and also why this suddenly became an issue.

Reproduction Steps

use a custom lambda asset like so:

 const statusLambda = new lambda.Function(this, 'StatusLambda', {
            code: lambda.Code.asset(path.join(__dirname, '../lambdas/codepipeline_status_lambda')),
            handler: 'index.handler',
            timeout: cdk.Duration.seconds(300),
            runtime: lambda.Runtime.NODEJS_10_X,
            environment: {
                ACCESS_TOKEN: cdk.SecretValue.secretsManager(props.gitHubSecretArn).toString(),
                DATADOG_API_KEY: cdk.SecretValue.secretsManager(props.dataDogSecretArn).toString(),
                ENV_TYPE: props.env_type,
                STAGE_TYPE: props.stage_type,
                APP_URL: props.url,
            },
        })

What did you expect to happen?

cdk diff to pass without error

What actually happened?

cdk diff returned this error:

unable to determine cloud assembly output directory. Assets must be defined indirectly within a "Stage" or an "App" scope
Subprocess exited with error 1

Verbose output:

CDK toolkit version: 1.62.0 (build 8c2d7fc)
Command line arguments: {
  _: [ 'diff' ],
  profile: 'outline-dev',
  v: 4,
  verbose: 4,
  'ignore-errors': false,
  ignoreErrors: false,
  json: false,
  j: false,
  ec2creds: undefined,
  i: undefined,
  'version-reporting': undefined,
  versionReporting: undefined,
  'path-metadata': true,
  pathMetadata: true,
  'asset-metadata': true,
  assetMetadata: true,
  'role-arn': undefined,
  r: undefined,
  roleArn: undefined,
  staging: true,
  'no-color': false,
  noColor: false,
  fail: false,
  'context-lines': 3,
  contextLines: 3,
  strict: false,
  '$0': 'cdk',
  STACKS: [ 'DevCoreStack' ],
  stacks: [ 'DevCoreStack' ]
}
cdk.json: {
  "app": "npx ts-node bin/corestack.ts"
}
cdk.context.json: {
  "availability-zones:account=349142687622:region=us-west-2": [
    "us-west-2a",
    "us-west-2b",
    "us-west-2c",
    "us-west-2d"
  ],
  "availability-zones:account=349142687622:region=us-east-1": [
    "us-east-1a",
    "us-east-1b",
    "us-east-1c",
    "us-east-1d",
    "us-east-1e",
    "us-east-1f"
  ],
  "availability-zones:account=703994937577:region=us-east-1": [
    "us-east-1a",
    "us-east-1b",
    "us-east-1c",
    "us-east-1d",
    "us-east-1e",
    "us-east-1f"
  ],
  "availability-zones:account=703994937577:region=us-east-2": [
    "us-east-2a",
    "us-east-2b",
    "us-east-2c"
  ]
}
merged settings: {
  versionReporting: true,
  pathMetadata: true,
  output: 'cdk.out',
  app: 'npx ts-node bin/corestack.ts',
  context: {},
  assetMetadata: true,
  profile: 'outline-dev',
  toolkitBucket: {},
  staging: true
}
Determining if we're on an EC2 instance.
Does not look like an EC2 instance.
Toolkit stack: CDKToolkit
Setting "CDK_DEFAULT_REGION" environment variable to us-east-1
Resolving default credentials
Retrieved account ID 349142687622 from disk cache
Setting "CDK_DEFAULT_ACCOUNT" environment variable to 349142687622
context: {
  'availability-zones:account=349142687622:region=us-west-2': [ 'us-west-2a', 'us-west-2b', 'us-west-2c', 'us-west-2d' ],
  'availability-zones:account=349142687622:region=us-east-1': [
    'us-east-1a',
    'us-east-1b',
    'us-east-1c',
    'us-east-1d',
    'us-east-1e',
    'us-east-1f'
  ],
  'availability-zones:account=703994937577:region=us-east-1': [
    'us-east-1a',
    'us-east-1b',
    'us-east-1c',
    'us-east-1d',
    'us-east-1e',
    'us-east-1f'
  ],
  'availability-zones:account=703994937577:region=us-east-2': [ 'us-east-2a', 'us-east-2b', 'us-east-2c' ],
  'aws:cdk:enable-path-metadata': true,
  'aws:cdk:enable-asset-metadata': true
}
outdir: cdk.out
env: {
  CDK_DEFAULT_REGION: 'us-east-1',
  CDK_DEFAULT_ACCOUNT: '349142687622',
  CDK_CONTEXT_JSON: '{"availability-zones:account=349142687622:region=us-west-2":["us-west-2a","us-west-2b","us-west-2c","us-west-2d"],"availability-zones:account=349142687622:region=us-east-1":["us-east-1a","us-east-1b","us-east-1c","us-east-1d","us-east-1e","us-east-1f"],"availability-zones:account=703994937577:region=us-east-1":["us-east-1a","us-east-1b","us-east-1c","us-east-1d","us-east-1e","us-east-1f"],"availability-zones:account=703994937577:region=us-east-2":["us-east-2a","us-east-2b","us-east-2c"],"aws:cdk:enable-path-metadata":true,"aws:cdk:enable-asset-metadata":true}',
  CDK_OUTDIR: 'cdk.out',
  CDK_CLI_ASM_VERSION: '5.0.0',
  CDK_CLI_VERSION: '1.62.0'
}
unable to determine cloud assembly output directory. Assets must be defined indirectly within a "Stage" or an "App" scope
Subprocess exited with error 1
Error: Subprocess exited with error 1
    at ChildProcess.<anonymous> (/Users/grahambooth/.nvm/versions/node/v12.18.3/lib/node_modules/aws-cdk/lib/api/cxapp/exec.ts:118:23)
    at ChildProcess.emit (events.js:315:20)
    at ChildProcess.EventEmitter.emit (domain.js:483:12)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:275:12)

Environment

• CLI Version : 1.62.0
• Framework Version:
• Node.js Version:v12.18.3
• OS : macOS
• Language (Version):TypeScript (3.8.3)

Other

---

This is 🐛 Bug Report

[mrgrain]

I think I’ve tracked this issue down.

As @jogold pointed out, it has to do with nesting different copies of @aws-cdk/core. However I politely disagree and would argue it is a very valid use case, especially with monorepos. It occurs whenever packages aren’t hoisted to the top. Lerna provides a mode for this and even npm-v7 workspaces seem to encourage this node_modules structure.

Generally speaking I don’t think it sits well with CDK to enforce a single instance of the code, not just the same version number.

Therefore the example repo @Shogan created is still a good starting point. The issue occurs with any kind of assets. Lambda Functions with Code Assets are an obvious one as they are often pulled out into a reusable library. However like @flochaz this also appears to happen when the code asset is just a parameter to a lambda lib. The reason being that a lambda with logRetention will cause assets for the custom resource to be stages, which will then fail.

---

Enough of the prelude. The issue surfaces because stages are enforced for asset staging as of 1.57.0. Specifically this change by @eladb

Now, Stages have been around for a long time. However they have been pretty much optional and I guess where only used with the new pipelines. Long story short, they use a different mechanic to determine if a thing is a stage object.

The Stage class uses instanceof which will return false for objects that are not from the same code instance.

All other “global” packages use a mechanism with Symbol.for: App, Stack, Aspect

---

Unless there is a good reason for this difference, I’d suggest to use the same mechanism for Stages which will fix this issue. If someone could confirm the direction, I’m happy to provide a PR as it is holding us back.

I’d also suggest to include a test for nested code copies.

[mrgrain]

We’ve just updated a bunch of our lerna stacks to 1.72.0 and it finally works again! 🎉