(aws-msk): CloudFormation error enabling open monitoring setting to MSK cluster

What is the problem?

When trying to add openMonitoring to an existing MSK cluster in an existing stack, CloudFormation applies the change to the cluster, but fails applying the change to the CloudFormation stack and rolls back the change in the stack, the resource keeps with the change.

Reproduction Steps

Add openMonitoring configuration to an existing MSK cluster with the following:

...
      openMonitoring: {
        prometheus: {
          jmxExporter: {
            enabledInBroker: true
          },
          nodeExporter: {
            enabledInBroker: true
          }
        }
      }
...

What did you expect to happen?

Changes are applied to the resource and the CloudFormation stack without errors.

What actually happened?

CloudFormation emits error after updating the MSK cluster configuration:

1:54:48 PM | UPDATE_FAILED        | AWS::MSK::Cluster   | MSKCluster
Amazon CloudFormation wasn't able to complete the request because of an issue with the server. Try again later.

Change is applied to the resource, but the stack it’s rolled back, so any further try to apply any change to that stack always detect the previous change (the stack is drifted) is not applied and tries to apply it again.

CDK CLI Version

1.129.0 (build fb43f89)

Framework Version

No response

Node.js Version

v16.3.0

OS

MacOS

Language

Typescript

Language Version

Typescript 3.9.7

Other information

MSK cluster is defined with L1 construct. Cluster Kafka version is 2.2.1.

Also tried to:

• Add enhancedMonitoring: _msk.ClusterMonitoringLevel.PER_TOPIC_PER_PARTITION configuration with the same result
• Upgrading cdk version to latest (1.132.0) with the same result

Didn’t try:

• Apply directly CloudFormation template

More info:

$ cdk diff my-stack

Stack my-stack
Resources
[~] AWS::MSK::Cluster MSKCluster MSKCluster
 ├─ [+] EnhancedMonitoring
 │   └─ PER_TOPIC_PER_PARTITION
 └─ [+] OpenMonitoring
     └─ {"Prometheus":{"JmxExporter":{"EnabledInBroker":true},"NodeExporter":{"EnabledInBroker":true}}}

$ cdk deploy my-stack

my-stack: deploying...
my-stack: creating CloudFormation changeset...
1:54:48 PM | UPDATE_FAILED        | AWS::MSK::Cluster   | MSKCluster
Amazon CloudFormation wasn't able to complete the request because of an issue with the server. Try again later.

	new MSKStack (/my-project-location/my-app/lib/msk.ts:63:24)
	\_ /my-project-location/my-app/bin/my-app.ts:45:3
	\_ Array.forEach (<anonymous>)
	\_ Object.<anonymous> (/my-project-location/my-app/bin/my-app.ts:21:6)
	\_ Module._compile (node:internal/modules/cjs/loader:1109:14)
	\_ Module.m._compile (/my-project-location/my-app/node_modules/ts-node/src/index.ts:1056:23)
	\_ Module._extensions..js (node:internal/modules/cjs/loader:1138:10)
	\_ Object.require.extensions.<computed> [as .ts] (/my-project-location/my-app/node_modules/ts-node/src/index.ts:1059:12)
	\_ Module.load (node:internal/modules/cjs/loader:989:32)
	\_ Function.Module._load (node:internal/modules/cjs/loader:829:14)
	\_ Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:76:12)
	\_ main (/my-project-location/my-app/node_modules/ts-node/src/bin.ts:198:14)
	\_ Object.<anonymous> (/my-project-location/my-app/node_modules/ts-node/src/bin.ts:288:3)
	\_ Module._compile (node:internal/modules/cjs/loader:1109:14)
	\_ Object.Module._extensions..js (node:internal/modules/cjs/loader:1138:10)
	\_ Module.load (node:internal/modules/cjs/loader:989:32)
	\_ Function.Module._load (node:internal/modules/cjs/loader:829:14)
	\_ Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:76:12)
	\_ node:internal/main/run_main_module:17:47


 ❌  my-stack failed: Error: The stack named my-stack failed to deploy: UPDATE_ROLLBACK_COMPLETE
    at Object.waitForStackDeploy (/usr/local/lib/node_modules/aws-cdk/lib/api/util/cloudformation.ts:307:11)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at prepareAndExecuteChangeSet (/usr/local/lib/node_modules/aws-cdk/lib/api/deploy-stack.ts:352:26)
    at CdkToolkit.deploy (/usr/local/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:189:24)
    at initCommandLine (/usr/local/lib/node_modules/aws-cdk/bin/cdk.ts:225:9)
The stack named my-stack failed to deploy: UPDATE_ROLLBACK_COMPLETE

[neovasili]

Thanks for the info 😁

I want to make a PoC repeating the create from CFN and migrate to CDK using the “raw” method I used and using the one you mentioned to see if there is any difference in the behavior, will be back as soon as I can do it

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.