Serverless Function Tracing Needs an Isolated Scope

• Review the documentation: https://docs.sentry.io/
• Search for existing issues: https://github.com/getsentry/sentry-javascript/issues
• Use the latest release: https://github.com/getsentry/sentry-javascript/releases
• Provide a link to the affected event from your Sentry account: https://sentry.io/organizations/par-tech-inc/performance/tdp-log-processing:248e161ab69049b29769dbbe61ed5da6/?transaction=process_generator_sessions

Package + Version

• @sentry/node 6.3.6

Version:

6.3.6

Description

Here is my withSentry wrapper for Azure serverless Function invocations:

const Sentry = require("@sentry/node");
const Tracing = require("@sentry/tracing");
const mongodb = require("mongodb");

const config = require("../../config/sentry");

if (config.dsn) {
    Sentry.init({
        ...config,
        integrations: [
            // enable DB calls tracing
            new Tracing.Integrations.Mongo({ useMongoose: true })
            // new Tracing.Integrations.Mysql()  // No integration available for mysql2.
        ]
    });
}

/**
 * Add Sentry transaction information to outgoing messages.
 * @param {*} transaction
 * @param {*} results
 * @returns
 */
const propagateTransaction = (transaction, results) => {
    if (results && transaction) {
        for (const message of Object.values(results)) {
            Object.assign(message, {
                sentryParentSpanId: transaction.spanId,
                sentryTraceId: transaction.traceId,
                sentrySampled: transaction.sampled
            });
        }
    }
    return results;
};

/**
 * A higher order function which logs error data to Sentry.
 *
 * @param {*} callback          The actual function which might throw errors.
 */
module.exports = callback => async (context, trigger, ...args) => {
    let transaction;
    if (config.dsn) {
        // Define a transaction for performance tracking.
        const transactionContext = {
            op: "serverless-task",
            name: context?.executionContext?.functionName || "test-function"
        };
        if ("sentryParentSpanId" in trigger) {
            Object.assign(transactionContext, {
                parentSpanId: trigger.sentryParentSpanId,
                traceId: trigger.sentryTraceId,
                sampled: trigger.sentrySampled
            });
        }
        transaction = Sentry.startTransaction(transactionContext);

        Sentry.configureScope(scope => {
            scope.setSpan(transaction);
            scope.setContext(
                "executionContext",
                context?.executionContext || {
                    errorMessage:
                        "No executionContext defined.  This could be a notification from a test environment."
                }
            );
            scope.setContext("functionTrigger", trigger);
        });
    }

    try {
        return propagateTransaction(
            transaction,
            await callback(context, trigger, ...args)
        );
    } catch (error) {
        if (config.dsn) {
            Sentry.captureException(error, scope => {
                scope.setContext(
                    "executionContext",
                    context?.executionContext || {
                        errorMessage:
                            "No executionContext defined.  This could be a notification from a test environment."
                    }
                );
                scope.setContext("functionTrigger", trigger);
                scope.setContext(
                    "theFullContext",
                    Object.assign(
                        {},
                        ...Object.keys(context).map(k => ({
                            [k]: JSON.stringify(context[k])
                        }))
                    )
                );
            });
            scope.setSpan(transaction);
            await Sentry.flush();
        }

        throw error;
    } finally {
        await transaction?.finish();
    }
};

Defining an Azure Function like this:

module.exports = withSentry(processLogFile);

This logs everything correctly for exceptions since the scope is set immediately before sending the log, but performance tracing associates DB children with the wrong span and other context. This appears to be because the DB integrations use the span set on the scope. Since we have a high volume application (on the order of 3400 invocations per minute), many function invocations run concurrently and, since Sentry.configureScope() configures the span in a global scope, logged child DB transactions are associated not with the invocation they actually ran during but instead with context of the function invocation that last called Sentry.configureScope(). In these images, there are dozens of MongoDB calls associated with the *_sessions function even though the inserts are actually all called by the parent *_timeseries function:

[derekrprice]

@richardsimko, check out #4071 for a detailed explanation of why you may not have experienced this issue yet with your web app. Basically, the problem is hidden in low traffic scenarios.

[mattfysh]

Could this not also be solved the same way the http handler works? i.e. running the wrapped handler in a new domain?