Adding provisioned concurrency to AWS Lambda results in temporary "Internal server error"

I have a lambda function that did not have any provisioned concurrency. When I added provisionedConcurrency: 10 to my serverless.yml file and deployed, my function resulted in a “Internal server error” response. API Gateway showed the following error:

The API with ID {id} doesn’t include a resource with path /* having an integration {function_arn} on the ANY method.

After the deploy was finished and the provisioned concurrency was ready, the error stopped. If I change the provisioned concurrency from 10 to 5, for example, I don’t receive any error, so the issue seems to be related to adding provisioned capacity to a function that didn’t already have it.

serverless.yml

provider:
  name: aws
  runtime: nodejs12.x
  apiGateway:
    binaryMediaTypes:
      - '*/*'
  stage: dev
  region: ap-northeast-1
functions:
  app:
    handler: index.handler
    timeout: 10
    events:
      - http: ANY /
      - http: 'ANY {proxy+}'
    provisionedConcurrency: 10

⚠️⚠️ REPLACE WITH FULL COMMAND NAME output

> serverless deploy -v

Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service {my_app}.zip file to S3 (68.49 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
CloudFormation - UPDATE_IN_PROGRESS - AWS::CloudFormation::Stack - {my_app}
CloudFormation - UPDATE_IN_PROGRESS - AWS::Lambda::Function - AppLambdaFunction
CloudFormation - UPDATE_COMPLETE - AWS::Lambda::Function - AppLambdaFunction
CloudFormation - UPDATE_IN_PROGRESS - AWS::ApiGateway::Method - ApiGatewayMethodAny
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Alias - AppProvConcLambdaAlias
CloudFormation - UPDATE_IN_PROGRESS - AWS::ApiGateway::Method - ApiGatewayMethodProxyVarAny
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Alias - AppProvConcLambdaAlias
CloudFormation - UPDATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethodAny
CloudFormation - UPDATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethodProxyVarAny
CloudFormation - CREATE_IN_PROGRESS - AWS::ApiGateway::Deployment - ApiGatewayDeployment{ID}
CloudFormation - CREATE_IN_PROGRESS - AWS::ApiGateway::Deployment - ApiGatewayDeployment{ID}
CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Deployment - ApiGatewayDeployment{ID}
CloudFormation - CREATE_COMPLETE - AWS::Lambda::Alias - AppProvConcLambdaAlias
CloudFormation - UPDATE_IN_PROGRESS - AWS::Lambda::Permission - AppLambdaPermissionApiGateway
CloudFormation - UPDATE_IN_PROGRESS - AWS::Lambda::Permission - AppLambdaPermissionApiGateway
CloudFormation - UPDATE_COMPLETE - AWS::Lambda::Permission - AppLambdaPermissionApiGateway
CloudFormation - UPDATE_COMPLETE_CLEANUP_IN_PROGRESS - AWS::CloudFormation::Stack - {my_app}
CloudFormation - DELETE_IN_PROGRESS - AWS::Lambda::Permission - AppLambdaPermissionApiGateway
CloudFormation - DELETE_IN_PROGRESS - AWS::ApiGateway::Deployment - ApiGatewayDeployment{ID}
CloudFormation - DELETE_COMPLETE - AWS::ApiGateway::Deployment - ApiGatewayDeployment{ID}
CloudFormation - DELETE_COMPLETE - AWS::Lambda::Permission - AppLambdaPermissionApiGateway
CloudFormation - UPDATE_COMPLETE - AWS::CloudFormation::Stack - {my_app}

Installed version

Framework Core: 1.67.0
Plugin: 3.6.1
SDK: 2.3.0
Components: 2.22.3

[burketyler]

We had this issue occur during a production deployment, which was completely unexpected as you can perform the same action in the console with no downtime. I haven’t traced the exact series of events that leads to this occurring but it appears to be a timing issue during the deploy where either 1) API Gateway has doesn’t yet have permissions to invoke the newly created alias, or 2) the lambda code has not been completely deployed before API Gateway has been switched to route to the new alias.

Regardless of whether this is caused by Serverless or CloudFormation I would have expected an issue that can cause downtime during an sls deploy to be properly documented, which doesn’t appear to be the case. How has something like this not been addressed in over a year?

[wolfejw86]

Looks like there are several gotchas related to updating a lambda with provisioned concurrency. While not about serverless framework, I found this article helpful since I have just now run into the above issue as well. It helped me understand the pieces at play better - posting here in case it helps someone else find their zero downtime solution faster.

https://futurice.com/blog/building-an-aws-serverless-system-provisioned-concurrency-zero-downtime