Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

sam local start-api slow to start when we have many lambdas and layers

See original GitHub issue

Description:

I’m not sure if this a bug or something that needs to be treated as feature. We are using sam since version 0.50.0 and we decided to move to the latest version 1.46.0. Our sam template has about 85 lambda definitions and 2 lambda layers that are being attached to pretty much all the 85 lambdas. We never had issues starting the local api by using version 0.50.0, it was pretty fast. When we updated to the latest version it takes almost 2 minutes in my computer to kick off everything. We have identified that the issue is with the lambda layers because when we remove all the references of the lambda layers from the lambda functions the api is starting within a couple of seconds. Running the sam local start-api --debug, we can see that there is something like a strange iteration/recursion that is happening, which seems that is like scanning all the resources again for each layer that we attach in each lambda.

We have identified that this is the flow: The _locate_layer_from_ref function is being called in the sam_function_provider.py and then this function is being called layer_resource = stack.resources.get(layer_logical_id)

Tracing this function in the sam_stack_provider.py, there is an iteration in the get function line 77 and the get_all is being invoked which does another iteration and probably if we trace it even more there might be one more. That’s the reason that make the whole bootstrap of the local api so slow. I don’t really understand what is the approach behind this implementation so if someone can have a look and maybe try to improve the logic will be really helpful. Or if someone can guide me about a better approach then I can try to do an improvement

Steps to reproduce:

I have created a simple template that can be used to reproduce the problem. The only thing that you need to do is to comment out the section that is referring the lambda layer within each lambda and then start the api. Then you’ll realise the difference in the bootstrap time

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  checker-api
  
Globals:
  Function:
    Timeout: 30

Resources:
  ApiGateway:
    Type: AWS::Serverless::Api
    Properties:
      StageName: 'dev'
      TracingEnabled: true
      Cors:
        AllowHeaders: "'authorization, token-id, Content-Type, Access-Control-Allow-Headers, Authorization, X-Requested-With'"
        AllowOrigin: "'*'"

  InternalDependencyLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: sam-app-dependencies
      Description: Dependencies for sam app [temp-units-conv]
      ContentUri: dependencies/
      CompatibleRuntimes:
        - nodejs14.x
      LicenseInfo: 'MIT'
      RetentionPolicy: Retain

  WordProcessFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/word-process/
      Handler: index.wordProcess
      Runtime: nodejs14.x
      MemorySize: 512
      Layers:
        - !Ref InternalDependencyLayer
      Architectures:
        - x86_64
      Events:
        WordProcess:
          Type: Api
          Properties:
            Path: /word-process
            Method: post
            RestApiId: !Ref ApiGateway

  WordProcessFunction1:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/word-process/
      Handler: index.wordProcess
      Runtime: nodejs14.x
      MemorySize: 512
      Layers:
        - !Ref InternalDependencyLayer
      Architectures:
        - x86_64
      Events:
        WordProcess:
          Type: Api
          Properties:
            Path: /word-process-1
            Method: post
            RestApiId: !Ref ApiGateway

  WordProcessFunction2:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/word-process/
      Handler: index.wordProcess
      Runtime: nodejs14.x
      MemorySize: 512
      Layers:
        - !Ref InternalDependencyLayer
      Architectures:
        - x86_64
      Events:
        WordProcess:
          Type: Api
          Properties:
            Path: /word-process-2
            Method: post
            RestApiId: !Ref ApiGateway

  WordProcessFunction3:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/word-process/
      Handler: index.wordProcess
      Runtime: nodejs14.x
      MemorySize: 512
      Layers:
        - !Ref InternalDependencyLayer
      Architectures:
        - x86_64
      Events:
        WordProcess:
          Type: Api
          Properties:
            Path: /word-process-3
            Method: post
            RestApiId: !Ref ApiGateway

  WordProcessFunction4:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/word-process/
      Handler: index.wordProcess
      Runtime: nodejs14.x
      MemorySize: 512
      Layers:
        - !Ref InternalDependencyLayer
      Architectures:
        - x86_64
      Events:
        WordProcess:
          Type: Api
          Properties:
            Path: /word-process-4
            Method: post
            RestApiId: !Ref ApiGateway

  WordProcessFunction5:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/word-process/
      Handler: index.wordProcess
      Runtime: nodejs14.x
      MemorySize: 512
      Layers:
        - !Ref InternalDependencyLayer
      Architectures:
        - x86_64
      Events:
        WordProcess:
          Type: Api
          Properties:
            Path: /word-process-5
            Method: post
            RestApiId: !Ref ApiGateway

Outputs:
  WordProcessFunction:
    Description: "Hello World Lambda Function ARN"
    Value: !GetAtt WordProcessFunction.Arn
  WordProcessFunctionIamRole:
    Description: "Implicit IAM Role created for Hello World function"
    Value: !GetAtt WordProcessFunctionRole.Arn

Observed result:

Start the api with sam local start-api --debug and you’ll realise that there are significantly more logs and whenever there are more lambdas in the file with the attached layer it can take minutes to start the api.