Performance: filter spans based on duration

Using the APM, my span index is full of db spans (span.type:db) containing lots of fast execution queries that are of little interest. A method to filter these spans based on duration will help to ignore fast queries and trace only important long running queries. Using sampling can ignore important long queries.

I suggest a dynamic configuration option based on span type and subtype: span_min_duration_[type]_[subtype]=[ms]

Example:

• span_min_duration_db=500ms will record only spans with type db and any subtype longer than 500ms
• span_min_duration_app_maintenance=100ms will record spans with type app and subtype maintenance longer than 100ms

Related: https://github.com/elastic/apm-agent-java/issues/440

[cdalexndr]

My example scales, in a 2GB span index with 11.5M docs, 95% spans are db type and more than 99% of db span duration is below 4ms. Such a waste of disk space…

Thinking to disable db spans and use postgres logging feature of threshold duration queries, then parse with logstash and manual crossreference log time with transaction to find the source, until this issue is resolved.

[SylvainJuge]

Also related, there is another related upcoming feature to handle fast exit spans (which include DB spans):

• At APM product level : https://github.com/elastic/apm/issues/496
• At Java-agent level : https://github.com/elastic/apm-agent-java/issues/2083