async hook stack has become corrupted

Hello,

I met strange error after I added sentry to my project. When the error is thrown, the project crashes immediatly.

this:

 Error: async hook stack has become corrupted (actual: 286, expected: 0)

or this:

 Error: async hook stack has become corrupted (actual: 908, expected: 0)

sometimes this:

 node[1]: ../src/api/callback.cc:125:void node::InternalCallbackScope::Close(): Assertion `(env_->execution_async_id()) == (0)' failed.

I know it comes with sentry because I tried to remove Sentry.init and the error disappears. In my project, I’m using the async_hooks API and I fear a conflict with your tracing SDK. I am using async_hooks this way:

const { AsyncLocalStorage } = require('async_hooks');
const store = new AsyncLocalStorage();

// ...

app.use((req, res, next) => {
  store.enterWith(new Map());

  const instance = store.getStore();
  instance.set('request-id', uuid());

  return next();
});

I am declaring Sentry this way

Sentry.init({
    environment: process.env.NODE_ENV,
    dsn: '__',
    integrations: [
      // enable HTTP calls tracing
      new Sentry.Integrations.Http({ tracing: true }),
      // enable Express.js middleware tracing
      new Tracing.Integrations.Express({ app: router }),
    ],

    // Set tracesSampleRate to 1.0 to capture 100%
    // of transactions for performance monitoring.
    // We recommend adjusting this value in production
    tracesSampleRate: 1.0,
  });

  // RequestHandler creates a separate execution context using domains, so that every
  // transaction/span/breadcrumb is attached to its own Hub instance
  router.use(Sentry.Handlers.requestHandler());
  // TracingHandler creates a trace for every incoming request
  router.use(Sentry.Handlers.tracingHandler());

Do you know how this issue could be solved ?

Thank you for your work 😃

Version

Node: 16 (Using Docker image node:16-alpine)

Packages

@sentry/node: 6.4.1 @sentry/tracing: 6.4.1

[Hongbo-Miao]

Thanks @kamilogorek! I am aware of I don’t have OpenTelemetry server in this demo repo. I am using this as a demo to reproduce the bug. Yeah just confirmed it only started from 16.2.0. Opened a ticket on Node.js side at https://github.com/nodejs/node/issues/38814.

[kamilogorek]

This is some kind of regression in Node v16.2.0. It’s working fine in v16.1.0.

The problem however, is incorrect request rejections handling between domains and async_hooks. Even simplest domain.create().run(() => next()) as an express middleware, when used with async_hooks (which are used in opentelemetry for context management - https://github.com/open-telemetry/opentelemetry-js/blob/dfb77372d311587c3462a43497c2ec4b3867e9fa/packages/opentelemetry-node/src/NodeTracerProvider.ts#L61-L63)` will throw an error. For some strange reason, when used together, they cannot talk with each other correctly.

It’s most likely that error thrown within async_hooks context is not correctly propagated to the domain, and it crashes the process instead of being handled there. Might be related:

https://github.com/nodejs/node/issues/38503 https://github.com/nodejs/node/issues/38145

One additional information here though, is that what’s causing an actual error that is crashing the process, is TCP not being able to connect to the OpenTelemetry server, which you can see when using Node v16.1.0

{"stack":"Error: connect ECONNREFUSED 127.0.0.1:55681\n    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1133:16)\n    at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:131:17)","message":"connect ECONNREFUSED 127.0.0.1:55681","errno":"-61","code":"ECONNREFUSED","syscall":"connect","address":"127.0.0.1","port":"55681","name":"Error"}

55681 is default port for OpenTelemetry, and because you are using CollectorTraceExporter, and not providing url override, it’ll try to send request to localhost:55681, which will reject, and effectively fail and crash the process.

There’s nothing that we can do from our side right now, as it’s infrastructure specific issue, not the SDK itself. We need to wait for Node to state clearly if it’s going to be fixed, and if not, then update docs appropriately.