Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

cdk deploy failed due to: ValidationError "TemplateURL must reference a valid S3 object to which you have access."

See original GitHub issue

We run cdk deploy in our continuous deployment pipeline. Today, we observed a failure due to “Error [ValidationError]: TemplateURL must reference a valid S3 object to which you have access.”. We are having cdk deploy create the template and upload to S3. We have only seen this trigger once in at least 100 builds, and we have not been able to reliably reproduce. I’m wondering if there is a race condition.

Looking at the source code, the deployStack() method where this error is thrown is marked experimental.

it looks like the template would be uploaded with publishAssets(): from:

but for some reason that didn’t finish, or failed, or used the wrong value and the s3 object wasn’t available when createChangeSet() was called here:

could also be that we hit some slow eventual consistency with the s3 object being available.

Reproduction Steps

cdk deploy seems to fail very rarely.

Error Log

13:56:22  yarn run cdk deploy lock-service \
13:56:22  	--role-arn=arn:aws:iam::ACCOUNT_ID:role/serverless_deployment_role \
13:56:22  	--context ENV_NAME=foo \
13:56:22  	--context BRANCH_NAME=bar \
13:56:22  	--context ACCOUNT=ACCOUNT_ID
13:56:22  yarn run v1.17.3
13:56:22  $ ./node_modules/.bin/cdk deploy lock-service --role-arn=arn:aws:iam::ACCOUNT_ID:role/serverless_deployment_role --context ENV_NAME=foo --context BRANCH_NAME=bar --context ACCOUNT=ACCOUNT_ID
13:56:28  {
13:56:28    context: _98point6Context {
13:56:28      env: 'foo',
13:56:28      service: 'lock-service',
13:56:28      branch: 'bar',
13:56:28      region: 'us-west-2',
13:56:28      account: 'ACCOUNT_ID',
13:56:28      config: {
13:56:28        apiAlarms: [Object],
13:56:28        client: [Object],
13:56:28        dynamoAlarms: [Object],
13:56:28        elasticsearch: [Object],
13:56:28        lambdaAlarms: [Object],
13:56:28        logging: [Object],
13:56:28        vpc: [Object]
13:56:28      }
13:56:28    },
13:56:28    disableAlarms: undefined
13:56:28  }
13:56:28 lock-service-bar: deploying...
13:56:28 lock-service-bar: creating CloudFormation changeset...
13:56:29   ❌ lock-service-bar failed: Error [ValidationError]: TemplateURL must reference a valid S3 object to which you have access.
13:56:29      at Request.extractError (./node_modules/aws-cdk/node_modules/aws-sdk/lib/protocol/query.js:50:29)
13:56:29      at Request.callListeners (./node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
13:56:29      at Request.emit (./node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
13:56:29      at Request.emit (./node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:683:14)
13:56:29      at Request.transition (./node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:22:10)
13:56:29      at AcceptorStateMachine.runTo (./node_modules/aws-cdk/node_modules/aws-sdk/lib/state_machine.js:14:12)
13:56:29      at ./node_modules/aws-cdk/node_modules/aws-sdk/lib/state_machine.js:26:10
13:56:29      at Request.<anonymous> (./node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:38:9)
13:56:29      at Request.<anonymous> (./node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:685:12)
13:56:29      at Request.callListeners (./node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
13:56:29    message: 'TemplateURL must reference a valid S3 object to which you have access.',
13:56:29    code: 'ValidationError',
13:56:29    time: 2020-05-13T20:56:29.180Z,
13:56:29    requestId: '97bf<redacted>',
13:56:29    statusCode: 400,
13:56:29    retryable: false,
13:56:29    retryDelay: 247.8930735018098
13:56:29  }
13:56:29  TemplateURL must reference a valid S3 object to which you have access.
13:56:29  error Command failed with exit code 1.


  • **CLI Version :1.22.0 (build 309ac1b)
  • **Framework Version:1.31.0
  • **OS :linux
  • **Language :typescript


This is 🐛 Bug Report

Issue Analytics

  • State:open
  • Created 3 years ago
  • Comments:7 (2 by maintainers)

github_iconTop GitHub Comments

zahirulducommented, Mar 9, 2022

I am facing a same problem when trying to deploy.

 ❌  dev-user-service-infra-cognito failed: Error [ValidationError]: Stack:arn:aws:cloudformation:us-east-1:701751291959:stack/dev-user-service-infra-cognito/5ae67480-4e9c-11ec-a035-12f6d51fe9c7 is in UPDATE_ROLLBACK_FAILED state and can not be updated.
    at Request.extractError (C:\Users\Asus\AppData\Roaming\nvm\v16.12.0\node_modules\aws-sdk\lib\protocol\query.js:50:29)
    at Request.callListeners (C:\Users\Asus\AppData\Roaming\nvm\v16.12.0\node_modules\aws-sdk\lib\sequential_executor.js:106:20)
    at Request.emit (C:\Users\Asus\AppData\Roaming\nvm\v16.12.0\node_modules\aws-sdk\lib\sequential_executor.js:78:10)
    at Request.emit (C:\Users\Asus\AppData\Roaming\nvm\v16.12.0\node_modules\aws-sdk\lib\request.js:686:14)
    at Request.transition (C:\Users\Asus\AppData\Roaming\nvm\v16.12.0\node_modules\aws-sdk\lib\request.js:22:10)
    at AcceptorStateMachine.runTo (C:\Users\Asus\AppData\Roaming\nvm\v16.12.0\node_modules\aws-sdk\lib\state_machine.js:14:12)
    at C:\Users\Asus\AppData\Roaming\nvm\v16.12.0\node_modules\aws-sdk\lib\state_machine.js:26:10
    at Request.<anonymous> (C:\Users\Asus\AppData\Roaming\nvm\v16.12.0\node_modules\aws-sdk\lib\request.js:38:9)
    at Request.<anonymous> (C:\Users\Asus\AppData\Roaming\nvm\v16.12.0\node_modules\aws-sdk\lib\request.js:688:12)
    at Request.callListeners (C:\Users\Asus\AppData\Roaming\nvm\v16.12.0\node_modules\aws-sdk\lib\sequential_executor.js:116:18) {
  code: 'ValidationError',
  time: 2022-03-09T05:55:04.101Z,
  requestId: '9f52811b-97dd-45ff-b1e3-f8522168c1f4',
  statusCode: 400,
  retryable: false,
  retryDelay: 802.838290909253
roryj-vendiacommented, Jul 25, 2022

Just ran into this issue myself when using the CDK. This is the first time it happened for us, but it did lead to a good amount of confusion.

Perhaps we could add a retry or two to verify that the s3 object is available before proceeding with the deployment?

This seems like a good approach. The CDK manages the template, deploying the template to S3, and calling CloudFormation to create changesets. Having a retry to handle the case when there is an issue due to eventual consistency seems like a nice customer-centric solution.

I can’t tell if this issue occurred during a timeframe where S3 was experiencing any issues, but that could also be a cause for the behaviour you ran into.

This event happened for us on 2022-07-20T18:46:13.607947Z in us-east-1. It did not appear to be an S3 issue, as all of our other uses of S3 seemed to be ok at the time, and there was no other indication on the service health page or personal dashboard

Read more comments on GitHub >

github_iconTop Results From Across the Web

TemplateURL must reference a valid S3 object to which you ...
amazon s3 - TemplateURL must reference a valid S3 object to which you have access - Stack Overflow. Collectives™ on Stack Overflow – ......
Read more >
Resolve template validation or template format errors in ...
This returns the following error: "Template validation error: Invalid template property or properties [Bucket]." This error is caused because ...
Read more >
CloudFormation — Boto3 Docs 1.26.36 documentation
CloudFormation generates the change set by comparing this template with the stack that you specified. Conditional: You must specify only TemplateBody or ...
Read more >
How do I resolve template validation or template format errors ...
For more details see the Knowledge Center article with this video: ...
Read more >
awslabs/aws-cdk - Gitter
i create restAPI with deploy:true and some resources in stack 1; i create the remaining resources in ... fails because the resources need...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found