elastic-apm-node with async hooks enabled leaks Transactions when used with cls-rtracer

elastic-apm-node leaks Transaction objects when it is used in an express app that uses require("cls_rtracer").expressMiddleware().

To Reproduce

Steps to reproduce the behavior:

1. Unzip cls-rtracer-example.zip
2. Cd to cls-rtracer-example directory
3. Run npm ci
4. Run ELASTIC_APM_SERVICE_NAME=cls-rtracer-example ELASTIC_APM_SERVER_URL=http://elastic-server:8200 node --require elastic-apm-node/start index.js
5. Hit http://localhost:3000 a bunch of time in a browser or with curl

Expected behavior

Following these steps the activeTransactions map in node_modules/elastic-apm-node/lib/instrumentation/async-hooks.js will continue to grow when the server processes http requests until the node process eventually runs out of memory.

Environment (please complete the following information)

• OS: Fedora Linux 31 x86_64
• Node.js version: Reproduced with 12.18.3 and 14.7.0
• APM Server version: 8.5
• Agent version: 3.6.1

How are you starting the agent? (please tick one of the boxes)

• Calling agent.start() directly (e.g. require('elastic-apm-node').start(...))
• Requiring elastic-apm-node/start from within the source code
• Starting node with -r elastic-apm-node/start

Additional context

I believe it is important to use cls-rtracer 2.x to reproduce this bug as it uses the AsyncLocalStorage API that is part of the async_hooks module in recent versions of node.

• Agent config options

  Click to expand

  ELASTIC_APM_SERVICE_NAME=cls-rtracer-example
  ELASTIC_APM_SERVER_URL=http://elastic-server:8200
  

• package.json dependencies:

  Click to expand

  "dependencies": {
      "cls-rtracer": "^2.2.0",
      "elastic-apm-node": "^3.6.1",
      "express": "^4.17.1"
  }
  

[puzpuzpuz]

@tsondergaard I’ve checked the reproducer and it seems that current wrapHttpEmitters implementation in cls-rtracer (which monkey-patches EE.emit() method) conflicts with elastic-apm-node CLS related logic. As the result not all Transaction objects are removed from the activeTransactions map. I’ve changed the logic, so that the problem should be gone once this PR passes review: https://github.com/puzpuzpuz/cls-rtracer/pull/39

Thanks for reporting this issue.

[puzpuzpuz]

@tsondergaard I see now. Going to take a look at the reproducer tomorrow.