elastic apm does not work with follow-redirects@0.2.0

One of my dependencies uses follow-redirects@0.2.0 elastic apm will hand request when follow-redirect version 0.2.0 (never version works, but may not able to upgrade).

require('elastic-apm-node').start({
    serviceName: 'example-app-local',
    secretToken: '<token>',
    usePathAsTransactionName: true,
    active: true,
    serverUrl:
      '<url>'
  })

const { http } = require('follow-redirects');
// const http = require('http')
const express = require('express')
const app = express()
const port = 3000

const url = 'http://bit.ly/UHfDGO'

app.get('/test', (req, res) => {
  let body = ''
// this will hang
  http.get(url, response => {
    response.on('data', chunk => {
      console.log(chunk.toString())
      body += chunk.toString();
    })
    .on('end', () => {
      console.log('end')
      res.send(body)
    })
  })
  .on('error', err => {
    console.error(err);
  });
  
})

app.listen(port, () => {
  console.log(`example app listening at http://localhost:${port}`)
})

[astorm]

I’m worried about what this means for the elastic apm library.

@msluther I was able to reproduce the hanging behavior described in the original issue. We’re happy to reopen this and talk through the issue with you. In general the guiding principle of all Elastic’s APM agents is to not break existing functionality whenever it’s possible and reasonable not to.

Re: wherever it’s possible or reasonable not to – there’s often some cases where “what’s possible/reasonable” is open to interpretation, and this is one of them. Our approach in these cases is to work with users and customers to make sure they get a solution that works. Sometimes that’s recommending they upgrade to a supported version of a problematic library, and other times that’s digging into a issue in order to fix a crasher/hanger.

The TL;DR; on this one is this seems like a bug that came out of the changes that follow-redirect makes the Node’s request behavior. I’m going to give you some background on our agent, and then get into why this is happening with follow redirect.

Elastic Agent’s HTTP

Here’s what the elastic agent does with regards to outgoing requests. We monkey patch the request and get methods of both the http and https modules. You can see us monkey patching the methods here

https://github.com/elastic/apm-agent-nodejs/blob/a07eb3a6616d25bf91cb0cf4ce982d7c1af18be4/lib/instrumentation/modules/http.js#L22-L25

https://github.com/elastic/apm-agent-nodejs/blob/a07eb3a6616d25bf91cb0cf4ce982d7c1af18be4/lib/instrumentation/modules/https.js#L23-L27

and here’s the code that runs in place of the request/get methods

https://github.com/elastic/apm-agent-nodejs/blob/a07eb3a6616d25bf91cb0cf4ce982d7c1af18be4/lib/instrumentation/http-shared.js#L131

The basic philosophy of our all wrappers is to

1. Perform whatever instrumentation we need to do
2. Once done, call the original method with its arguments

We’re generally not trying to change the implementation of these methods, we (like most/all APM agents) just need to perform additional actions when methods are called.

Follow Redirect at 0.2.0

The follow-redirects module (at version 0.0.2) works by returning objects that extend the http and https modules and redefines how the get and request methods work. In v0.2.0 this extending is is done via prototype swapping (this approach is abandoned in later versions). Unlike the Node.js agent, follow-redirects changes the implementation of these methods significantly. (which isn’t intended as a negative critique – changing the implementation is the point of this module).

I spent a bit of time debugging their old code, and as best I can tell, when the elastic Node.js Agent is active, the technique that follow-redirect uses the create a requestProxy object here doesn’t work.

https://github.com/follow-redirects/follow-redirects/blob/9a13ff409ce5b05463ee0785cb4c048c21800561/create.js#L30-L35

		var requestProxy = Object.create(clientRequest);
		requestProxy._events = {};
		requestProxy._eventsCount = 0;
		if (callback) {
			requestProxy.on('response', callback);
		}

The callback in the above code is the callback registered to handle the request. While it’s registered with the requestProxy object’s response event, the response event is never invoked. Because it’s never invoked, the request hangs. Looking at this code, it’s a little unclear how this could ever work universally – even with the agent disabled. In fact, if we dig back into the follow-redirect repo and look at this PR, it appears that this technique has known issues

The technique used in b4554ff to proxy a request seems to introduce a problem in which connections are not properly closed. This pull requests adds a separate object to act as a proxy of native requests, which as an added benefit brings abort() should abort the whole chain

So, it sounds like under certain circumstances, follow-redirect@0.2.0 will fail to close the connection. If the connection never closes, the response event never fires and the callbacks never fire. That sounds like the behavior that was reported in this issue. The PR fix was added to version 0.3.0 of follow-redirect. The behavior described in this bug is not present in follow-redirects@0.3.0.

So, to sum up again, this seems like an old bug in the follow-redirect module that the agent happens to trigger – not because the agent is doing anything wrong, but because the technique that follow-redirect@0.2.0 used was unreliable AND follow-redirect@0.2.0 is also changing the implentation of core Node.js methods. This would usually be on the authors of follow-redirect to fix (which they did!)

Given that this happened in a pre-1.0 module, and that it’s also been fixed, and the it was an acknowledged bug in the third party module, and the fix has been available for almost five years, this is probably something we won’t address.

Also, it’s not 100% clear what we could do about this case – the follow-redirect modules is failing to close the request.

What do you think @msluther – is there more we can/should do in this specific case? It’s always tricky when two modules are trying to rewrite the same bit of behavior, but in this case the follow-redirect@0.2.0 code seems to be the culprit.

[astorm]

Always happy to help @msluther – direct customer feedback is what makes the agent better over time. Best wishes rolling the agent out across your projects! Re-closing this one.