Enabling XRay tracing on an existing stack fails

Bug Report

Description

When enabling AWS X-Ray tracing on an existing stack, the update fails with the following error:

An error occurred: HelloLambdaFunction - The provided execution role does not have permissions to call PutTraceSegments on XRAY (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: ********-****-****-****-************).

This error does not seem to be reproducible 100% of the time and appears to be a race to update the Lambda Execution role and the function itself.

These CloudFormation logs show the IamRoleUpdate commenced after the function configuration update:

1. What did you do?

   • Deploy a Serverless stack with tracing disable
   • Update provider.tracing.lambda to true
   • Re-deploy the stack

2. What happened?

   • The error The provided execution role does not have permissions to call PutTraceSegments on XRAY is returned and the stack does not update

3. What should’ve happened?

   • The stack updates, grants the necessary permissions to the execution role, and enables Lambda X-Ray tracing

4. What’s the content of your serverless.yml file?

   • Initial stack creation:

   service:
     name: xray-permissions
   
   provider:
     name: aws
     runtime: nodejs10.x
   
   functions:
     hello:
       handler: handler.hello
       events:
         - http:
             method: get
             path: hello
   

   • Update:

   service:
     name: xray-permissions
   
   provider:
     name: aws
     runtime: nodejs10.x
     tracing:
       lambda: true # This has been enabled after creation
   
   functions:
     hello:
       handler: handler.hello
       events:
         - http:
             method: get
             path: hello
   

5. What’s the output you get when you use the SLS_DEBUG=* environment variable (e.g. SLS_DEBUG=* serverless deploy)

   • I didn’t manage to capture this debug information but will update the information when I can reproduce the issue reliably.

Similar or dependent issues:

• #3495

[medikoo]

@jamesmbourne thanks for report. Technically this can happen when you provide a custom execution role (via provider.role or via role directly in a function configuration).

Can you confirm it’s not the case?

As, if default role is used, exactly those permissions are set: https://github.com/serverless/serverless/blob/55c259f994b0062d60c64407f0bec6f17ef44d57/lib/plugins/aws/package/compile/functions/index.js#L319-L332

[medikoo]

@bisoldi it means that it needs to be reported at plugin repository, as it’s an issue over there