Instrumentation sometimes breaks outgoing HTTP requests

Describe the bug

We’ve been hunting down multiple weird bugs recently, all of them related to outgoing HTTP requests. After some trial-and-error I found it’s most likely caused by APM’s instrumentation as disabling it fixed all issues.

Example of our code:

got('https://registry.npmjs.org/uikit').then((response) => {
    console.log(JSON.parse(response.body));
});

This sometimes throws an error:

Unexpected token  in JSON at position 0 in "https://registry.npmjs.org/uikit": 
    ��T�n�6���MQw=hh��/���H��\K�JR�E���d;N�h���I<�3�����@

The response, which looks completely broken at first, is actually correct, except it’s gzipped - npm responds with content-encoding: gzip header and the got module is supposed to gunzip the response and pass the result to our callback. For some reason, sometimes it passes the raw response instead.

At first I suspected this was an issue with got, so I tried downgrading to an older version which we’ve been using for a while without any problems but that didn’t help. Then I tried to disable instrumentation and as I already said, that fixed all issues. I went through the code of this module but didn’t find any got-specific code, so I’m not sure if this only affects that module or all outgoing requests.

Unfortunately, I was not able to reliably reproduce this locally. In production, we get the error roughly for one in a hundred requests and usually when a request for the same URL is made a few seconds later, it succeeds. It is also not specific to the host being npmjs.org or response format being json - we fetch data and files from multiple different services and it affects all of them.

Let me know if there’s anything I can do to get this problem identified and fixed.

Environment (please complete the following information)

• OS: Linux
• Node.js version: 8.11.3
• APM Server version: 6.3.0
• Agent version: 1.8.1

How are you starting the agent? (please tick one of the boxes)

• Calling agent.start() directly (e.g. require('elastic-apm-node').start(...))
• Requiring elastic-apm-node/start from within the source code
• Starting node with -r elastic-apm-node/start

Additional context Add any other context about the problem here.

• Agent config options

  Click to expand

      active: process.env.NODE_ENV === 'production',
      serviceName: 'jsdelivr-origin',
      serviceVersion: require('../package.json').version,
      logLevel: 'fatal',
      captureSpanStackTraces: false,
      captureErrorLogStackTraces: 'always',
      errorOnAbortedRequests: true,
      abortedErrorThreshold: 30000,
      transactionSampleRate: .2,
      asyncHooks: false, // had this enabled but tried to disable - didn't help
  

• package.json dependencies:

  Click to expand

  	"@octokit/rest": "^15.9.3",
  	"aws-sdk": "^2.265.1",
  	"bluebird": "^3.5.1",
  	"bunyan": "^1.8.12",
  	"bunyan-prettystream": "0.1.3",
  	"bytes": "^3.0.0",
  	"clean-css": "^4.1.11",
  	"combine-source-map": "0.8.0",
  	"compression": "^1.7.2",
  	"config": "^1.30.0",
  	"config-mapper-env": "^1.0.1",
  	"connect-modrewrite": "^0.10.2",
  	"convert-source-map": "^1.5.1",
  	"dedent-js": "~1.0",
  	"elastic-apm-node": "^1.8.1",
  	"elastic-apm-utils": "^1.1.0",
  	"elasticsearch": "^15.0.0",
  	"express": "^4.16.3",
  	"fs-extra": "^6.0.1",
  	"got": "^8.3.1",
  	"gunzip-maybe": "^1.4.1",
  	"h-logger2": "^1.0.2",
  	"h-logger2-elastic": "^2.1.3",
  	"handlebars": "^4.0.11",
  	"is-hexdigest": "^1.0.1",
  	"is-minified-code": "^1.1.0",
  	"is-safe-path": "~1.0",
  	"lodash": "^4.17.10",
  	"lru-cache": "^4.1.3",
  	"mime": "^2.3.1",
  	"morgan": "^1.9.0",
  	"natural-compare": "~1.4",
  	"promisepipe": "^2.1.3",
  	"queue": "^4.4.2",
  	"recursive-readdir": "^2.2.2",
  	"semver": "^5.5.0",
  	"serve-favicon": "^2.5.0",
  	"tar-fs": "^1.16.3",
  	"through2-spy": "~2.0",
  	"uglify-js": "^3.3.23",
  	"write-file-atomic": "^2.3.0"
  

[watson]

This turned out to be an issue relate to mimic-response which is used by decompress-response which in turn is used by got. I’ve submitted a PR that hopefully will get merged: https://github.com/sindresorhus/mimic-response/pull/1

In the meantime (or in case that PR cannot be merged for whatever reason), I’ve opened a PR in this repo that patches mimic-response so that it plays nice together with this agent: #429

[MartinKolarik]

Thanks for a quick fix!