Spans not shown in the UI when their parent is missing

First, note that I’m not sure if this is a bug in the UI or an issue with the approach suggest in the node.js docs, which is why I’m opening this issue here.

Describe the bug

We use a span filter to only store spans above a certain duration. The logic behind that is rather simple: most of the spans are very short and of little value. Storing only spans above a certain duration saves a lot of space in the ES. This approach has been suggested in the node.js agent repo (elastic/apm-agent-nodejs#1701) and is now included in the performance tuning tips.

We found, however, that often even long spans are missing from the APM UI (i.e., there would be a transaction but without spans), even though the spans are actually stored and possible to find via Discover.

I believe this happens when a short span is dropped by the filter, while it was set as a parent of some much longer span. The UI doesn’t show the long span, because it can’t find its parent (the following message suggests that):

Expected behavior

We need a way to filter out short spans. I’m not sure, however, if the UI can be changed to show spans with a missing parent, or if the filtering method needs to be smarter and for example replace the parent id when the real parent is dropped (this currently seems hard to do, without support directly in the agent lib).

Debug logs

Two captured spans. The first is dropped by our filter. The second is stored but doesn’t show up in the UI.

Click to expand

{
  id: '492b3e17825250d4',
  transaction_id: 'ae0b52e3a6c60f64',
  parent_id: 'ae0b52e3a6c60f64',
  trace_id: '413dfce475cad1107182651ea9abab46',
  name: 'EXEC',
  type: 'cache',
  subtype: 'redis',
  action: null,
  timestamp: 1626368299979725,
  duration: 1.616,
  context: {
    db: undefined,
    http: undefined,
    tags: undefined,
    destination: { service: [Object], address: '127.0.0.1', port: 6379 },
    message: undefined
  },
  stacktrace: undefined,
  sync: false,
  outcome: 'success',
  sample_rate: 1
}

{
  id: '6523f3b42ff3c563',
  transaction_id: 'ae0b52e3a6c60f64',
  parent_id: '492b3e17825250d4',
  trace_id: '413dfce475cad1107182651ea9abab46',
  name: 'GET registry.npmjs.org/jquery',
  type: 'external',
  subtype: 'http',
  action: 'GET',
  timestamp: 1626368299989391,
  duration: 108.439,
  context: {
    db: undefined,
    http: {
      method: 'GET',
      status_code: 200,
      url: 'https://registry.npmjs.org/jquery'
    },
    tags: undefined,
    destination: { service: [Object], address: 'registry.npmjs.org', port: 443 },
    message: undefined
  },
  stacktrace: undefined,
  sync: false,
  outcome: 'success',
  sample_rate: 1
}

[trentm]

How will the mentioned changes affect chained short calls

In that case (again when we’ve implemented it), “span compression” should help. Those similar and consecutive redis spans should get grouped into a single span that represents all of them. The “drop fast exit spans” issue discussed above is a part of a larger recently spec’d effort to have an answer for huge traces. Probably the best current starting point for reading is: https://github.com/elastic/apm/tree/master/specs/agents/handling-huge-traces

#2100 is the implementation issue for compressed spans for the Node.js APM agent.

[MartinKolarik]

I see, thanks. That may indeed be enough. Our main issue is that we have many short redis calls and they are often assigned as parents of the long HTTP calls.

One thing I’m still not sure about though. How will the mentioned changes affect chained short calls, e.g.:

redis.multi().set(rKey, value).get(rKey).exec()

The above results in four super-short chained spans - even if the last one is considered exit span and dropped, I’m not sure if the dropping is then meant to be applied recursively - i.e., what happens to the first three?