‼️ NOTICE: last modified date does not match with the last modified date of the retrieved parameters

Please add your +1 👍 to let us know you have encountered this

Status: INVESTIGATING

Overview:

Cloudformation recently deployed a change to allow dynamic references to ssm parameters within templates without needing to provide a version. This allows always referencing the latest value of a parameter within your template. This change had a bug that caused change set creation to fail when the parameter value was updated.

This breakage was caused by the references resolving to a version of the parameter that include a timestamp. Upon changeset creation, if the parameter has been updated, cloudformation sees the parameter name and the previously resolved timestamped version as different values and fails to update. They have since changed logic so that the strings resolved do not include timestamps, however if a template was successfully deployed during the time that the timestamp was being appended, a stack cannot be updated via changeset while referencing a dynamic parameter.

UPDATE:

It is clear that removing all dynamic ssm references doesn’t cause users to get unstuck. Users have to manually deploy their stacks without a changeset to get past this. That being the case, for users with lots of stacks updating each manually is a pain. We are considering adding an option to the CLI to skip a changeset if this error occurs during deploy.

Complete Error Message:

Parameters: [PARAMETER_NAME] last modified date does not match with the last modified date of the retrieved parameters.

Workaround:

Attempt the following steps in order as a workaround:

1. Upgrade both your CLI and all your @aws-cdk/xyz modules to v1.103.0 - This removes dynamic SSM references and goes back to referencing them via CFN parameters.

2. Delete your stack and redeploy - If you are unable to do so because this is a production stack, and upgrading to v1.103 didn’t fix the issue, please comment here with the following information and try number 3.

   • The name of the parameter causing the failure
   • An example of how you are referencing that parameter in code, IE through an enum value, manually specifying the param name etc.

3. Run cdk synth and then manually deploy this stack using the the aws cli or cfn console - This is not an ideal workaround for users who may have a lot of different stacks that need updating but in the meantime, this can unblock you if you need a stack deployed. We are working on figuring out another option that will allow skipping changeset creation via the aws cli.

Solution:

This issue only shows up for users who deployed stacks during a short period of time before cloudformation fixed some of these bugs. Because of that, it is hard for us to test ourselves if removal of dynamic param references (https://github.com/aws/aws-cdk/pull/14527) solves this. If it doesn’t, another solution mentioned here is to update your stack without creating a changeset. We may be able to provide an option in the CLI to skip a changeset if this error is encountered. Since we don’t have broken stacks to test this with, please let us know if this is or an additional fix is needed.

Related Issues:

https://github.com/aws/aws-cdk/issues/14467 https://github.com/aws/aws-cli/issues/6106

[AbdoNile]

I managed to find a fix for this with @Shogan . It is a “surgical” workaround , but will allow you to get the stack in an “updateable” state using CDK without recreating it.

1. get the new ssm parameter last modified date time stamp from aws console or using ssm cli get-parameter
2. convert the time into Unix format including milliseconds . I used https://time.lol for this. You should end up with something that looks like “1616080117000”
3. In AWS Console cloudformation screen , open the stack , click on “edit in designer”
4. the lower pan of the designed contains the template in YAML format
5. find and replace all instances of “”{{resolve:ssm:/ssm_parameter_name:1:OLD_TIME_STAMP}}" with “”{{resolve:ssm:/ssm_parameter_name:1:NEW UNIX_TIME_STAMP}}"
6. Save the template to a local YAML file to your computer
7. Close the Designer
8. From CloudFormation Console again , Click on Update Stack , Replace Existing stack , upload your template file that you downloaded from the designer.
9. Follow the wizard until the stack update is completed.

If all goes well , you should be able to deploy the stack using cdk going forward.

[FlorinAsavoaie]

Also getting this issue for obtainDefaultFluentBitECRImage from @aws-cdk/aws-ecs… This is really frustrating.