(aws-kinesis): fails to deploy on-demand stream

What is the problem?

Error from CloudFormation when using Kinesis Data Streams in on-demand mode with shardCount specified.

ShardCount is not expected when StreamMode=ON_DEMAND

Reproduction Steps

Create a new stream with a streamMode set.

new Stream(this, 'KinesisStream', {
    streamName: 'my-ingest-stream',
    shardCount: 1, // optional
    streamMode: StreamMode.ON_DEMAND,
});

What did you expect to happen?

Deploy a stream in on-demand mode, possibly informed by the shardCount for an initial scaling hint.

What actually happened?

Error from CloudFormation

ShardCount is not expected when StreamMode=ON_DEMAND

CDK CLI Version

1.137.0 (build bfbdf64)

Framework Version

1.137.0

Node.js Version

14.17.3

OS

MacOS

Language

Typescript

Language Version

4.5.2

Other information

CloudFormation’s documentation does not inform the user that setting the StreamMode to ON_DEMNAD is an error. Can we raise a bug with the right team to get the documentation improved?

Removing the shardCount field from the CDK user code does not fix the error because internally, CDK always sets the shardCount to 1 if it is left undefined by the user.

The solution in CDK is to remove shardCount from the underlying CfnStream if the user did not provide the field. CloudFormation will default to 1 anyway if needed, so CDK’s default is not doing anything useful here.

We should also add a validator to ensure that shardCount and StreamMode: 'ON_DEMAND' are not set at the same time when generating the template.

An end-to-end test through CloudFormation would have caught this bug before release. I’m not aware that there is any automated testing of this type at present.

[pierre-vr]

⚠️ Edit: Caution this might only work for creating a new stream from scratch (see comment below)

In the meantime, a quick fix is to manually set the shardCount prop to undefined on the Cfn resource:

const stream = new Stream(this, 'KinesisStream', {
    streamName: 'my-ingest-stream',
    streamMode: StreamMode.ON_DEMAND,
});

;(stream.node.defaultChild as CfnStream).shardCount = undefined

I have tested it and I am able to confirm that “ShardCount” is no longer present in the Cfn template.

[Harshit22]

Thanks @igilham for detailed workaround ! Upgrading to CDK version 1.138.2 fixed the issue.