Serverless Rate exceeded for Cloudformation API

Hi there! I have an issue connected with throttling on CloudFormation API.

Right now there is:

• 26 functions
• 3 dynamo tables, 1 sqs queue.
• sum: 160 cf resources

The issue occurs on the serverless deployment

  ServerlessError: An error occurred: 
BroadcastToClientsEventSourceMappingDynamodbChannelsMessagesTable - Rate exceeded for operation 'Rate exceeded (Service: Lambda, Status Code: 429, Request ID: b8b7796b-e9d7-4c84-9615-7d2a98c1c85b, Extended Request ID: null)
'..node_modules/serverless/lib/plugins/aws/lib/monitorStack.js:125:33
  From previous event:
      at AwsDeploy.monitorStack (node_modules/serverless/lib/plugins/aws/lib/monitorStack.js:28:12)
      at node_modules/serverless/lib/plugins/aws/lib/updateStack.js:115:28

I’m pasting cloudformation error. Its always the same on sqs and dynamodb stream events mapping.

This is my investigation:

1. CloudFormation resources limit ~160 - checked
2. Resources names limit (64) - I renamed to resources to the shortnames & error still occurs randomly - checked Core issue always comes from EventSourceMapping resource creation. ie. (BroadcastToClientsEventSourceMappingDynamodbChannelsMessagesTable - Rate exceeded for operation 'Rate exceeded (Service: Lambda, Status Code: 429)
3. Error occurs only on “new stack creation” - all resources are triggered to deploy and then cloudformation returns rate exceed response on EventSourceMapping resource - checked
4. Deploying new stack needs to comment 50% of the functions in the resources files - checked
5. Splitting plugin doesn’t help in type/group mode

serverless.yml

broadcastToClients:
  handler: bin/broadcastToClients
  package:
    include:
      - bin/broadcastToClients
  events:
    - stream:
        type: dynamodb
        batchSize: 1
        enabled: true
        startingPosition: LATEST
        arn:
          Fn::GetAtt:
            - channelsMessagesTable
            - StreamArn


channelBackup:
  handler: bin/channelBackup
  package:
    include:
      - bin/channelBackup
  events:
    - sqs:
        exist: true
        arn:
          Fn::GetAtt:
            - backupQueue
            - Arn

deploy command

sls deploy --verbose --stage xyz

 Node Version:              12.18.1
 Framework Version:         1.73.1
 Plugin Version:            3.6.14
 SDK Version:               2.3.1
 Components Version:        2.31.6

Installed version

1.73.1

[herebebogans]

I got an update from our TAM that the issue has been reported by a number of customers and is being investigated by the Lambda service team.

[adrianbanasiak]

These are not serverless throttling api call errors you are seeing - they are on the cloudformation AWS side. You will see simliar errors if you try to create to many concurrent Dynamo tables or Kinesis streams in a CF template.

Of course they ar not, it states in the error message that they are coming from AWS.

I am reporting that because in my opinion Serverless should be able to work around such cases or tell user what to check or modify in order to deploy its stack.