cdk deploy failed due to: ValidationError "TemplateURL must reference a valid S3 object to which you have access."

We run cdk deploy in our continuous deployment pipeline. Today, we observed a failure due to “Error [ValidationError]: TemplateURL must reference a valid S3 object to which you have access.”. We are having cdk deploy create the template and upload to S3. We have only seen this trigger once in at least 100 builds, and we have not been able to reliably reproduce. I’m wondering if there is a race condition.

Looking at the source code, the deployStack() method where this error is thrown is marked experimental.

it looks like the template would be uploaded with publishAssets(): https://github.com/aws/aws-cdk/blob/1a29e26336dbc536128f1a9279fc081998ff80bb/packages/aws-cdk/lib/util/asset-publishing.ts#L10 from: https://github.com/aws/aws-cdk/blob/1a29e26336dbc536128f1a9279fc081998ff80bb/packages/aws-cdk/lib/api/deploy-stack.ts#L196

but for some reason that didn’t finish, or failed, or used the wrong value and the s3 object wasn’t available when createChangeSet() was called here: https://github.com/aws/aws-cdk/blob/1a29e26336dbc536128f1a9279fc081998ff80bb/packages/aws-cdk/lib/api/deploy-stack.ts#L203

could also be that we hit some slow eventual consistency with the s3 object being available.

Reproduction Steps

cdk deploy seems to fail very rarely.

Error Log

13:56:22  yarn run cdk deploy lock-service \
13:56:22  	--role-arn=arn:aws:iam::ACCOUNT_ID:role/serverless_deployment_role \
13:56:22  	--context ENV_NAME=foo \
13:56:22  	--context BRANCH_NAME=bar \
13:56:22  	--context ACCOUNT=ACCOUNT_ID
13:56:22  yarn run v1.17.3
13:56:22  $ ./node_modules/.bin/cdk deploy lock-service --role-arn=arn:aws:iam::ACCOUNT_ID:role/serverless_deployment_role --context ENV_NAME=foo --context BRANCH_NAME=bar --context ACCOUNT=ACCOUNT_ID
13:56:28  {
13:56:28    context: _98point6Context {
13:56:28      env: 'foo',
13:56:28      service: 'lock-service',
13:56:28      branch: 'bar',
13:56:28      region: 'us-west-2',
13:56:28      account: 'ACCOUNT_ID',
13:56:28      config: {
13:56:28        apiAlarms: [Object],
13:56:28        client: [Object],
13:56:28        dynamoAlarms: [Object],
13:56:28        elasticsearch: [Object],
13:56:28        lambdaAlarms: [Object],
13:56:28        logging: [Object],
13:56:28        vpc: [Object]
13:56:28      }
13:56:28    },
13:56:28    disableAlarms: undefined
13:56:28  }
13:56:28 lock-service-bar: deploying...
13:56:28 lock-service-bar: creating CloudFormation changeset...
13:56:29  
13:56:29   ❌ lock-service-bar failed: Error [ValidationError]: TemplateURL must reference a valid S3 object to which you have access.
13:56:29      at Request.extractError (./node_modules/aws-cdk/node_modules/aws-sdk/lib/protocol/query.js:50:29)
13:56:29      at Request.callListeners (./node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
13:56:29      at Request.emit (./node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
13:56:29      at Request.emit (./node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:683:14)
13:56:29      at Request.transition (./node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:22:10)
13:56:29      at AcceptorStateMachine.runTo (./node_modules/aws-cdk/node_modules/aws-sdk/lib/state_machine.js:14:12)
13:56:29      at ./node_modules/aws-cdk/node_modules/aws-sdk/lib/state_machine.js:26:10
13:56:29      at Request.<anonymous> (./node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:38:9)
13:56:29      at Request.<anonymous> (./node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:685:12)
13:56:29      at Request.callListeners (./node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
13:56:29    message: 'TemplateURL must reference a valid S3 object to which you have access.',
13:56:29    code: 'ValidationError',
13:56:29    time: 2020-05-13T20:56:29.180Z,
13:56:29    requestId: '97bf<redacted>',
13:56:29    statusCode: 400,
13:56:29    retryable: false,
13:56:29    retryDelay: 247.8930735018098
13:56:29  }
13:56:29  TemplateURL must reference a valid S3 object to which you have access.
13:56:29  error Command failed with exit code 1.

Environment

• **CLI Version :1.22.0 (build 309ac1b)
• **Framework Version:1.31.0
• **OS :linux
• **Language :typescript

Other

---

This is 🐛 Bug Report

[zahiruldu]

I am facing a same problem when trying to deploy.

 ❌  dev-user-service-infra-cognito failed: Error [ValidationError]: Stack:arn:aws:cloudformation:us-east-1:701751291959:stack/dev-user-service-infra-cognito/5ae67480-4e9c-11ec-a035-12f6d51fe9c7 is in UPDATE_ROLLBACK_FAILED state and can not be updated.
    at Request.extractError (C:\Users\Asus\AppData\Roaming\nvm\v16.12.0\node_modules\aws-sdk\lib\protocol\query.js:50:29)
    at Request.callListeners (C:\Users\Asus\AppData\Roaming\nvm\v16.12.0\node_modules\aws-sdk\lib\sequential_executor.js:106:20)
    at Request.emit (C:\Users\Asus\AppData\Roaming\nvm\v16.12.0\node_modules\aws-sdk\lib\sequential_executor.js:78:10)
    at Request.emit (C:\Users\Asus\AppData\Roaming\nvm\v16.12.0\node_modules\aws-sdk\lib\request.js:686:14)
    at Request.transition (C:\Users\Asus\AppData\Roaming\nvm\v16.12.0\node_modules\aws-sdk\lib\request.js:22:10)
    at AcceptorStateMachine.runTo (C:\Users\Asus\AppData\Roaming\nvm\v16.12.0\node_modules\aws-sdk\lib\state_machine.js:14:12)
    at C:\Users\Asus\AppData\Roaming\nvm\v16.12.0\node_modules\aws-sdk\lib\state_machine.js:26:10
    at Request.<anonymous> (C:\Users\Asus\AppData\Roaming\nvm\v16.12.0\node_modules\aws-sdk\lib\request.js:38:9)
    at Request.<anonymous> (C:\Users\Asus\AppData\Roaming\nvm\v16.12.0\node_modules\aws-sdk\lib\request.js:688:12)
    at Request.callListeners (C:\Users\Asus\AppData\Roaming\nvm\v16.12.0\node_modules\aws-sdk\lib\sequential_executor.js:116:18) {
  code: 'ValidationError',
  time: 2022-03-09T05:55:04.101Z,
  requestId: '9f52811b-97dd-45ff-b1e3-f8522168c1f4',
  statusCode: 400,
  retryable: false,
  retryDelay: 802.838290909253
}

[roryj-vendia]

Just ran into this issue myself when using the CDK. This is the first time it happened for us, but it did lead to a good amount of confusion.

Perhaps we could add a retry or two to verify that the s3 object is available before proceeding with the deployment?

This seems like a good approach. The CDK manages the template, deploying the template to S3, and calling CloudFormation to create changesets. Having a retry to handle the case when there is an issue due to eventual consistency seems like a nice customer-centric solution.

I can’t tell if this issue occurred during a timeframe where S3 was experiencing any issues, but that could also be a cause for the behaviour you ran into.

This event happened for us on 2022-07-20T18:46:13.607947Z in us-east-1. It did not appear to be an S3 issue, as all of our other uses of S3 seemed to be ok at the time, and there was no other indication on the service health page or personal dashboard