APM Transactions not sampled when sample rate is 1
See original GitHub issueDescribe the bug:
We have micro services using Flask as framework and ElasticAPM for tracing requests. We noticed ~10% of transactions are not sampled despite not changing the default sample rate (1) for any of the services. Services send traces to APM servers running two different versions (7.14.1
and 7.16.1
). With APM server 7.14.1
, upsampled transactions have transaction.sampled
set to false
.
Environment
- Python version:
3.6
,3.7
- Framework and version:
Flask 1.0.2, 1.1.2, 2.0.2, 2.1.1
- APM Server version:
7.14.1
,7.16.1
- Agent version:
5.10.1
,6.7.2
Additional context The screenshot below shows upsampled transactions associated with one HTTP request:
transaction.sampled=false
transaction.span_count.started=0
Issue Analytics
- State:
- Created a year ago
- Comments:9 (3 by maintainers)
Top Results From Across the Web
Transaction sampling | APM User Guide [8.5] - Elastic
Distributed tracing and head based sampling example one. In this example, Service A initiates four transactions and has a sample rate of 1...
Read more >apm/tracing-sampling.md at main · elastic/apm - GitHub
1 means all transactions will be sampled (the default) · 0 means no transactions will be sampled · 0.5 means approximately 50% of...
Read more >Event limits and sampling for APM and mobile monitoring
A non-New Relic monitoring service may show different results from New Relic. Events that are capped and subject to sampling include: Transaction ......
Read more >Transaction Trace Sampling - AppOptics
However, in high-traffic production environments, it is possible to see sample rates of <1%. How do I know if a request has been...
Read more >FAQ | Leading Open-Source APM - GitHub Pages
There is a sampling rate option in the agent's pinpoint.config file (profiler.sampling.rate). Pinpoint agent samples 1 trace every N transactions if this ...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
You’ll want
trace_continuation_strategy=restart
ortrace_continuation_strategy=restart_external
, depending on whether the upstream service is Elastic-monitored. (You can tell if it’s an Elastic traceparent by checking thetracestate
header for anes=
entry.)Note, however, that this setting was only introduced in version 6.10.0 of the agent, and the versions you listed above are older than that.
Updating the service with
trace_continuation_strategy=restart
and ensuring the deployment is up to date fixes the issue. Thank you @basepi for your help in resolving this issue.