Help understanding why spans are being dropped

Hello, this is not a bug report but a help request. If this is not the right place I apologize in advance.

In the APM section of Kibana I see these warnings:

The APM agent that reported this transaction dropped X spans or more based on its configuration.

I’ve read the linked documentation and added the following environment vars:

- name: ELASTIC_APM_SPAN_FRAMES_MIN_DURATION
  value: "-1"
- name: ELASTIC_APM_TRANSACTION_MAX_SPANS
  value: "1000"
- name: ELASTIC_APM_STACK_TRACE_LIMIT
  value: "1000"

I’ve tried different values for ELASTIC_APM_TRANSACTION_MAX_SPANS: 1000, 2000 and 4000 but spans are still being dropped in the transactions.

My transactions are very simple, they have a dozen of spans at most. In another cluster I’ve running with less requests the spans are not being dropped.

How can I find the cause of the spans being dropped or the right value for ELASTIC_APM_TRANSACTION_MAX_SPANS?

If the cause were unforeseen errors like it says in the documentation what can I do to find them?

Thank you.

Environment

• OS: Debian 11 (Docker)
• Python version: 3.9.13
• Framework and version: Flask 2.1.3
• APM Server version: 7.12.1
• Agent version: 6.10.2

[basepi]

I haven’t forgotten you! The other dev for the python agent is coming back from leave next week, I’m going to pick his brain and see if he has any bright ideas about what’s going on here.

[basepi]

Yes! Glad we solved it. 🚀