span is not sent to APM server

I see SPAN data with NodeJs agent 1.14.5 but 2.6.0 is failing to capture same JS code stays same only APM agent version is changing

debug output from 1.14.5 version:

init span { id: 'c5231c49-ad8d-4c08-8c54-e35704096dc1' }
intercepted call to http.request { id: 'c5231c49-ad8d-4c08-8c54-e35704096dc1' }
request details: { protocol: 'https:',
  host: 'iapi.xyz.com',
  id: 'c5231c49-ad8d-4c08-8c54-e35704096dc1' }
start span { id: 'c5231c49-ad8d-4c08-8c54-e35704096dc1',
  name: 'GET iapi.xyz.com/xyz/xyz/../up',
  type: 'ext.http.http' }
intercepted http.ClientRequest response event { id: 'c5231c49-ad8d-4c08-8c54-e35704096dc1' }
setting transaction result { id: 'c5231c49-ad8d-4c08-8c54-e35704096dc1',
  result: 'HTTP 2xx' }
setting default transaction name: GET /up { id: 'c5231c49-ad8d-4c08-8c54-e35704096dc1' }
ended span { id: 'c5231c49-ad8d-4c08-8c54-e35704096dc1',
  name: 'GET iapi.xyz.com/xyz/xyz/../up',
  type: 'ext.http.http',
  truncated: true }
adding transaction to queue { id: 'c5231c49-ad8d-4c08-8c54-e35704096dc1' }
setting timer to flush queue: 4860ms
ended transaction { id: 'c5231c49-ad8d-4c08-8c54-e35704096dc1',
  type: 'request',
  result: 'HTTP 2xx',
  name: 'GET /up' }
flushing queue
sending transactions payload

debug from 2.6.0 version:

intercepted request event call to http.Server.prototype.emit
start trace { trans: '54768b21709fe11d',
  parent: undefined,
  trace: 'bc3d74b2436610ae1574d7604a0d4c68',
  name: null,
  type: null }
start span { span: '26737ff7e8c227eb',
  parent: '54768b21709fe11d',
  trace: 'bc3d74b2436610ae1574d7604a0d4c68',
  name: null,
  type: 'ext.http.http' }
intercepted call to http.request { id: '54768b21709fe11d' }
request details: { protocol: 'https:',
  host: 'iapi.xyz.com',
  id: '54768b21709fe11d' }
intercepted http.ClientRequest response event { id: '54768b21709fe11d' }
setting transaction result { trans: '54768b21709fe11d',
  parent: undefined,
  trace: 'bc3d74b2436610ae1574d7604a0d4c68',
  result: 'HTTP 2xx' }
setting default transaction name: GET /up { trans: '54768b21709fe11d',
  parent: undefined,
  trace: 'bc3d74b2436610ae1574d7604a0d4c68' }
sending transaction { trans: '54768b21709fe11d',
  trace: 'bc3d74b2436610ae1574d7604a0d4c68' }
ended transaction { trans: '54768b21709fe11d',
  parent: undefined,
  trace: 'bc3d74b2436610ae1574d7604a0d4c68',
  type: 'request',
  result: 'HTTP 2xx',
  name: 'GET /up' }

I’m happy to share more details just tell me what would help here.

[watson]

Turns out that up until Node.js v7, the IncomingMessage object would emit aborted if the underlying socket was closed

[watson]

I just used event-debug to see what was going on on both the request, response and the underlying socket. Here’s the result:

adding hook to Node.js module loader
shimming http@11.10.1 module
shimming http.Server.prototype.emit function
shimming http.request function
shimming http.ServerResponse.prototype.writeHead function
start trace { trans: '4e2f1e90fc90f4ce',
  parent: undefined,
  trace: '7746926dd756ae7a0db94c04ae70659b',
  name: undefined,
  type: undefined }
start span { span: 'a35df073808cec1e',
  parent: '4e2f1e90fc90f4ce',
  trace: '7746926dd756ae7a0db94c04ae70659b',
  name: null,
  type: 'ext.http.http' }
intercepted call to http.request { id: '4e2f1e90fc90f4ce' }
request details: { protocol: 'http:',
  host: 'example.com',
  id: '4e2f1e90fc90f4ce' }
ClientRequest: socket +0ms
ClientRequest: prefinish +3ms
Socket: resume +1ms
Socket: lookup +1ms
Socket: connect +526ms
ClientRequest: finish +1ms
Socket: ready +2ms
Socket: data +199ms
ClientRequest: response +4ms
200
intercepted http.ClientRequest response event { id: '4e2f1e90fc90f4ce' }
IncomingMessage: readable +4ms
Socket: readable +0ms
Socket: end +1ms
Socket: close +2ms
ClientRequest: close +1ms

As can be seen here the response (IncomingMessage) never emits anything else than readable. We currently expect it to emit end at some point. I think that’s a mistake. But I wonder why it’s not emitting anything else when the underlying socket closes. I would have expected it to emit close in that case.