Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Performance degradation in 2.21.0
See original GitHub issueDescribe the bug After installing 2.21.0, the CPU profile of a couple of our microservices went crazy, increasing significantly. On occasion a container running the service would start failing a large percentage of requests timing out calling downstream services and need to be terminated. Upon rolling back to 2.20.0, the performance returned to normal.
To Reproduce Steps to reproduce the behavior: Install Datadog.Tracing.Bundle 2.21.0
Expected behavior Updating to latest version should not completely break our applications.
Screenshots
Above is for an application where you can clearly see the impact of running with the impacted version. (graph is CPU utilization for the ECS service in log scale for clarity).
Runtime environment (please complete the following information):
- Instrumentation mode: Using
Datadog.Tracing.Bundleto be able to use both automatic and manual instrumentation. - Tracer version: 2.21.0
- OS: Ubuntu Linux running on ECS Docker container
- CLR: .NET 6
Additional context We originally saw this problem before switching to the bundle (we were previously downloading the .deb file and installing in the container and matching the version in code). I thought this may have been the issue based on previous issues I’ve encountered, so switched to using the bundle, but it was still an issue. In both services where we saw this issue, deploying a rollback of this library (and only this library) was enough to return it to a functioning state.
Issue Analytics
- State:
- Created 9 months ago
- Reactions:1
- Comments:30
Top Results From Across the Web
Top Related Medium Post
Top Related StackOverflow Question
Troubleshoot Live Code
Top Related Reddit Thread
Top Related Hackernoon Post
Top Related Tweet
Top Related Dev.to Post
Top Related Hashnode Post
@Recio I think that would be great if you can open a new ticket. I pinged someone from the ASM team to look at this.
@george-zubrienko I’m glad to hear that. I was defining a strategy to investigate 😃 out of curiosity, was it something in your code? machine? option enabled from our side ?
If anything happen new, do not hesitate.
@Recio Yeah the fix in 2.22 was for a deadlock on linux. Can you try check if
DD_APSEC_ENABLEDis not set or at least set to0? This should not be enabled by default. I hope it will restore your happiness and sleep schedule 😃