question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

APM Transactions not sampled when sample rate is 1

See original GitHub issue

Describe the bug: We have micro services using Flask as framework and ElasticAPM for tracing requests. We noticed ~10% of transactions are not sampled despite not changing the default sample rate (1) for any of the services. Services send traces to APM servers running two different versions (7.14.1 and 7.16.1). With APM server 7.14.1, upsampled transactions have transaction.sampled set to false.

Environment

  • Python version: 3.6, 3.7
  • Framework and version: Flask 1.0.2, 1.1.2, 2.0.2, 2.1.1
  • APM Server version: 7.14.1, 7.16.1
  • Agent version: 5.10.1, 6.7.2

Additional context The screenshot below shows upsampled transactions associated with one HTTP request:

  • transaction.sampled=false
  • transaction.span_count.started=0 Screen Shot 2022-08-31 at 7 59 17 PM

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:9 (3 by maintainers)

github_iconTop GitHub Comments

2reactions
basepicommented, Aug 31, 2022

You’ll want trace_continuation_strategy=restart or trace_continuation_strategy=restart_external, depending on whether the upstream service is Elastic-monitored. (You can tell if it’s an Elastic traceparent by checking the tracestate header for an es= entry.)

Note, however, that this setting was only introduced in version 6.10.0 of the agent, and the versions you listed above are older than that.

1reaction
Alshehcommented, Sep 7, 2022

Updating the service with trace_continuation_strategy=restart and ensuring the deployment is up to date fixes the issue. Thank you @basepi for your help in resolving this issue.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Transaction sampling | APM User Guide [8.5] - Elastic
Distributed tracing and head based sampling example one. In this example, Service A initiates four transactions and has a sample rate of 1...
Read more >
apm/tracing-sampling.md at main · elastic/apm - GitHub
1 means all transactions will be sampled (the default) · 0 means no transactions will be sampled · 0.5 means approximately 50% of...
Read more >
Event limits and sampling for APM and mobile monitoring
A non-New Relic monitoring service may show different results from New Relic. Events that are capped and subject to sampling include: Transaction ......
Read more >
Transaction Trace Sampling - AppOptics
However, in high-traffic production environments, it is possible to see sample rates of <1%. How do I know if a request has been...
Read more >
FAQ | Leading Open-Source APM - GitHub Pages
There is a sampling rate option in the agent's pinpoint.config file (profiler.sampling.rate). Pinpoint agent samples 1 trace every N transactions if this ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found