AWS Layer handler serious performance issue when application import certain libraries

How do you use Sentry?

Sentry Saas (sentry.io)

Version

1.10.1

Steps to Reproduce

1. Create empty aws lambda function

File: lambda_function.py

# import hubspot  # https://github.com/HubSpot/hubspot-api-python
def lambda_handler(event, context):
    pass

File: mysentry.py

# mysentry.py
# https://docs.sentry.io/platforms/python/guides/aws-lambda/
import os
import sentry_sdk
from sentry_sdk.integrations.aws_lambda import AwsLambdaIntegration

sentry_sdk.init(
    # dsn="https://[e-mail]/0",
    dsn=os.environ['SENTRY_DSN'],
    # https://github.com/getsentry/sentry-python/blob/a5ee1bd8c5b456704b9629fc430fb5203602f3c7/sentry_sdk/consts.py#L56
    environment=os.environ['SENTRY_ENVIRONMENT'],
    integrations=[
        AwsLambdaIntegration(),
    ],
    # Set traces_sample_rate to 1.0 to capture 100%
    # of transactions for performance monitoring.
    # We recommend adjusting this value in production,
    traces_sample_rate=float(os.environ['SENTRY_TRACES_SAMPLE_RATE']),
)
module_name, handler_fn_name = os.environ['SENTRY_INITIAL_HANDLER'].rsplit('.', 1)
import importlib
handler_module = importlib.import_module(module_name)
handler_fn = getattr(handler_module, handler_fn_name)

def handler(event, context):
    return handler_fn(event, context)

1. Add layer (2 versions tested, both have same issue) arn:aws:lambda:ap-southeast-2:943013980633:layer:SentryPythonServerlessSDK:24 arn:aws:lambda:ap-southeast-2:943013980633:layer:SentryPythonServerlessSDK:42

2. Set handler (3 handlers to compare performance) lambda_function.lambda_handler # default empty handler function sentry_sdk.integrations.init_serverless_sdk.sentry_lambda_handler # sentry handler to enable sentry mysentry.handler # this shows init sentry is ok, the problem is in layer implementation

3. Set env SENTRY_DSN=“https://[e-mail]/0” SENTRY_ENVIRONMENT=production SENTRY_INITIAL_HANDLER=lambda_function.lambda_handler SENTRY_TRACES_SAMPLE_RATE=0.1

4. Test following situations, total 6 combinations

• with/without import hubspot
• test each handler (with any layer, same result)

Expected Result

Sentry init should not take too long

Actual Result

The combination “import hubspot” + “handler=sentry_lambda_handler” takes ~30 seconds to finish with 128MB mem. 1GB runs faster (~3 seconds).

The combination “import hubspot” + “handler=lambda_handler” takes ~2.3 seconds to finish with 128MB mem. This shows the problem should not happend even with lowest mem.

The combination “import hubspot” + “handler=mysentry.handler” takes ~3.2 seconds to finish with 128MB mem. This shows enable sentry sdk doesn’t have the problem. The problem is in layer implementation.

The combination “no import” + “handler=lambda_handler” takes ~0.1 seconds with 128MB mem. This shows a baseline of performance.

[zhaowb]

@antonpirker no worries, I’m not blocked by this issue. I’m planning to create a layer my own with just pre-install sentry package and the mysentry.py as above, so it would be a drop-in replacement of the official layer. Since using sentry_sdk.init() is suggested in official doc, I suppose I won’t lose much function comparing to the official layer, except I need to maintain the package version by my own.

[antonpirker]

Thanks for this information!

Right now I do not have a clue what the problem could be that it takes almost 30 seconds to start when hubspot is installed.

Maybe hubspot is monkey patching the same things that Sentry is monkey patching. But to verify this we need to read the code of hubspot.

I have put this on our backlog, but currently we have a lot on our plates. So I am not sure when I will have time.