Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Infinite Loop in datadog.trace.instrumentation.rabbitmq.amqp.TracedDelegatingConsumer
See original GitHub issueWe have an issue where our rabbit messages that are destined for mysql are failing due to a data violation. Under normal circumstances these will go through a 3x retry cycle and then bail out. However when we have APM enabled the TracedDelegatingConsumer seems to “get in the way” and either eat the exception or augment the underlying default spring behavior and the message is infinitely stuck in a retry loop.
We have turned off the APM tool and this behavior goes away.
The exception thrown that SHOULD be causing the item to stop retrying is o.s.a.r.r.RejectAndDontRequeueRecoverer : Retries exhausted for message. However we the below is also added and looped infinitely when APM is enabled below.
2018-11-27 15:50:04.660-06:00 ERROR- [pool-1-thread-26] o.s.a.r.c.CachingConnectionFactory : Channel shutdown: clean channel shutdown; protocol method: #method<channel.close>(reply-code=200, reply-text=Closed due to exception from Consumer datadog.trace.instrumentation.rabbitmq.amqp.TracedDelegatingConsumer@1b993fb7 (amq.ctag-ztJhz2pW_o919UV5_ltoSg) method handleDelivery for channel AMQChannel(amqp://[scrubbed]/[scrubbed],64773), class-id=0, method-id=0)
--
| 2018-11-27 15:50:04.658-06:00 ERROR- [pool-1-thread-26] c.r.c.impl.ForgivingExceptionHandler : Consumer datadog.trace.instrumentation.rabbitmq.amqp.TracedDelegatingConsumer@1b993fb7 (amq.ctag-ztJhz2pW_o919UV5_ltoSg) method handleDelivery for channel AMQChannel(amqp://[scrubbed]/[scrubbed],64773) threw an exception for channel AMQChannel(amqp://[scrubbed]/[scrubbed],64773) java.lang.NullPointerException: null
| 2018-11-27 15:50:04.598-06:00 ERROR- [pool-1-thread-16] o.s.a.r.c.CachingConnectionFactory : Channel shutdown: clean channel shutdown; protocol method: #method<channel.close>(reply-code=200, reply-text=Closed due to exception from Consumer datadog.trace.instrumentation.rabbitmq.amqp.TracedDelegatingConsumer@70bc6d0e (amq.ctag-Mn3tr6a-Dxt8hiUgma9EyA) method handleDelivery for channel AMQChannel(amqp://[scrubbed]/[scrubbed],64773), class-id=0, method-id=0)
| 2018-11-27 15:50:04.598-06:00 ERROR- [pool-1-thread-16] c.r.c.impl.ForgivingExceptionHandler : Consumer datadog.trace.instrumentation.rabbitmq.amqp.TracedDelegatingConsumer@70bc6d0e (amq.ctag-Mn3tr6a-Dxt8hiUgma9EyA) method handleDelivery for channel AMQChannel(amqp://[scrubbed]/[scrubbed],64773) threw an exception for channel AMQChannel(amqp://[scrubbed]/[scrubbed],64773) java.lang.NullPointerException: null
| 2018-11-27 15:50:04.595-06:00 ERROR- [pool-1-thread-8] o.s.a.r.c.CachingConnectionFactory : Channel shutdown: clean channel shutdown; protocol method: #method<channel.close>(reply-code=200, reply-text=Closed due to exception from Consumer datadog.trace.instrumentation.rabbitmq.amqp.TracedDelegatingConsumer@308fc027 (amq.ctag-7S5wgXceOldEEvYJ0_vPgg) method handleDelivery for channel AMQChannel(amqp://[scrubbed]/[scrubbed],64772), class-id=0, method-id=0)
| 2018-11-27 15:50:04.593-06:00 ERROR- [pool-1-thread-8] c.r.c.impl.ForgivingExceptionHandler : Consumer datadog.trace.instrumentation.rabbitmq.amqp.TracedDelegatingConsumer@308fc027 (amq.ctag-7S5wgXceOldEEvYJ0_vPgg) method handleDelivery for channel AMQChannel(amqp://[scrubbed]/[scrubbed],64772) threw an exception for channel AMQChannel(amqp://[scrubbed]/[scrubbed],64772) java.lang.NullPointerException: null
| 2018-11-27 15:50:04.564-06:00 ERROR- [pool-1-thread-18] o.s.a.r.c.CachingConnectionFactory : Channel shutdown: clean channel shutdown; protocol method: #method<channel.close>(reply-code=200, reply-text=Closed due to exception from Consumer datadog.trace.instrumentation.rabbitmq.amqp.TracedDelegatingConsumer@5fd4299 (amq.ctag-jBQGiGL6uPi4BEdD0gu-KQ) method handleDelivery for channel AMQChannel(amqp://[scrubbed]/[scrubbed],64772), class-id=0, method-id=0)
| 2018-11-27 15:50:04.563-06:00 ERROR- [pool-1-thread-18] c.r.c.impl.ForgivingExceptionHandler : Consumer datadog.trace.instrumentation.rabbitmq.amqp.TracedDelegatingConsumer@5fd4299 (amq.ctag-jBQGiGL6uPi4BEdD0gu-KQ) method handleDelivery for channel AMQChannel(amqp://[scrubbed]/[scrubbed],64772) threw an exception for channel AMQChannel(amqp://[scrubbed]/[scrubbed],64772) java.lang.NullPointerException: null
| 2018-11-27 15:50:04.498-06:00 ERROR- [pool-1-thread-25] o.s.a.r.c.CachingConnectionFactory : Channel shutdown: clean channel shutdown; protocol method: #method<channel.close>(reply-code=200, reply-text=Closed due to exception from Consumer datadog.trace.instrumentation.rabbitmq.amqp.TracedDelegatingConsumer@6dc1b602 (amq.ctag-eVXMUgZzirJkyYG2fhgwFg) method handleDelivery for channel AMQChannel(amqp://[scrubbed]/[scrubbed],64772), class-id=0, method-id=0)
| 2018-11-27 15:50:04.496-06:00 ERROR- [pool-1-thread-25] c.r.c.impl.ForgivingExceptionHandler : Consumer datadog.trace.instrumentation.rabbitmq.amqp.TracedDelegatingConsumer@6dc1b602 (amq.ctag-eVXMUgZzirJkyYG2fhgwFg) method handleDelivery for channel AMQChannel(amqp://[scrubbed]/[scrubbed],64772) threw an exception for channel AMQChannel(amqp://[scrubbed]/[scrubbed],64772) java.lang.NullPointerException: null
| 2018-11-27 15:50:04.496-06:00 ERROR- [pool-1-thread-7] o.s.a.r.c.CachingConnectionFactory : Channel shutdown: clean channel shutdown; protocol method: #method<channel.close>(reply-code=200, reply-text=Closed due to exception from Consumer datadog.trace.instrumentation.rabbitmq.amqp.TracedDelegatingConsumer@3c212de3 (amq.ctag-xPibB-_KrfwniVU83-0Q-g) method handleDelivery for channel AMQChannel(amqp://[scrubbed]/[scrubbed],64771), class-id=0, method-id=0)
| 2018-11-27 15:50:04.494-06:00 ERROR- [pool-1-thread-7] c.r.c.impl.ForgivingExceptionHandler : Consumer datadog.trace.instrumentation.rabbitmq.amqp.TracedDelegatingConsumer@3c212de3 (amq.ctag-xPibB-_KrfwniVU83-0Q-g) method handleDelivery for channel AMQChannel(amqp://[scrubbed]/[scrubbed],64771) threw an exception for channel AMQChannel(amqp://[scrubbed]/[scrubbed],64771) java.lang.NullPointerException: null
| 2018-11-27 15:50:04.471-06:00 ERROR- [pool-1-thread-20] o.s.a.r.c.CachingConnectionFactory : Channel shutdown: clean channel shutdown; protocol method: #method<channel.close>(reply-code=200, reply-text=Closed due to exception from Consumer datadog.trace.instrumentation.rabbitmq.amqp.TracedDelegatingConsumer@7b014a3d (amq.ctag-nbFCrYuc-__Dc2RFRrGRhw) method handleDelivery for channel AMQChannel(amqp://[scrubbed]/[scrubbed],64770), class-id=0, method-id=0)
| 2018-11-27 15:50:04.469-06:00 ERROR- [pool-1-thread-20] c.r.c.impl.ForgivingExceptionHandler : Consumer datadog.trace.instrumentation.rabbitmq.amqp.TracedDelegatingConsumer@7b014a3d (amq.ctag-nbFCrYuc-__Dc2RFRrGRhw) method handleDelivery for channel AMQChannel(amqp://[scrubbed]/[scrubbed],64770) threw an exception for channel AMQChannel(amqp://[scrubbed]/[scrubbed],64770) java.lang.NullPointerException: null
| 2018-11-27 15:50:04.391-06:00 ERROR- [pool-1-thread-12] o.s.a.r.c.CachingConnectionFactory : Channel shutdown: clean channel shutdown; protocol method: #method<channel.close>(reply-code=200, reply-text=Closed due to exception from Consumer datadog.trace.instrumentation.rabbitmq.amqp.TracedDelegatingConsumer@42e69391 (amq.ctag-5nJKe6c17JZdKdvUc5s1pA) method handleDelivery for channel AMQChannel(amqp://[scrubbed]/[scrubbed],64769), class-id=0, method-id=0)
| 2018-11-27 15:50:04.390-06:00 ERROR- [pool-1-thread-12] c.r.c.impl.ForgivingExceptionHandler : Consumer datadog.trace.instrumentation.rabbitmq.amqp.TracedDelegatingConsumer@42e69391 (amq.ctag-5nJKe6c17JZdKdvUc5s1pA) method handleDelivery for channel AMQChannel(amqp://[scrubbed]/[scrubbed],64769) threw an exception for channel AMQChannel(amqp://[scrubbed]/[scrubbed],64769) java.lang.NullPointerException: null
| 2018-11-27 15:50:04.390-06:00 ERROR- [pool-1-thread-14] o.s.a.r.c.CachingConnectionFactory : Channel shutdown: clean channel shutdown; protocol method: #method<channel.close>(reply-code=200, reply-text=Closed due to exception from Consumer datadog.trace.instrumentation.rabbitmq.amqp.TracedDelegatingConsumer@1ce46f8 (amq.ctag-b9RPANn6PE3f05BCZ5YYYA) method handleDelivery for channel AMQChannel(amqp://[scrubbed]/[scrubbed],64770), class-id=0, method-id=0)
| 2018-11-27 15:50:04.388-06:00 ERROR- [pool-1-thread-14] c.r.c.impl.ForgivingExceptionHandler : Consumer datadog.trace.instrumentation.rabbitmq.amqp.TracedDelegatingConsumer@1ce46f8 (amq.ctag-b9RPANn6PE3f05BCZ5YYYA) method handleDelivery for channel AMQChannel(amqp://[scrubbed]/[scrubbed],64770) threw an exception for channel AMQChannel(amqp://[scrubbed]/[scrubbed],64770) java.lang.NullPointerException: null
| 2018-11-27 15:50:04.375-06:00 ERROR- [pool-1-thread-16] o.s.a.r.c.CachingConnectionFactory : Channel shutdown: clean channel shutdown; protocol method: #method<channel.close>(reply-code=200, reply-text=Closed due to exception from Consumer datadog.trace.instrumentation.rabbitmq.amqp.TracedDelegatingConsumer@235112d9 (amq.ctag-464kxmffh3vP6XdYzkjRjQ) method handleDelivery for channel AMQChannel(amqp://[scrubbed]/[scrubbed],64768), class-id=0, method-id=0)
| 2018-11-27 15:50:04.374-06:00 ERROR- [pool-1-thread-16] c.r.c.impl.ForgivingExceptionHandler : Consumer datadog.trace.instrumentation.rabbitmq.amqp.TracedDelegatingConsumer@235112d9 (amq.ctag-464kxmffh3vP6XdYzkjRjQ) method handleDelivery for channel AMQChannel(amqp://[scrubbed]/[scrubbed],64768) threw an exception for channel AMQChannel(amqp://[scrubbed]/[scrubbed],64768) java.lang.NullPointerException: null
Issue Analytics
- State:
- Created 5 years ago
- Comments:14 (6 by maintainers)
Top Results From Across the Web
RabbitMQ - Datadog Docs
This check monitors RabbitMQ through the Datadog Agent. It allows you to: Track queue-based stats: queue size, consumer count, unacknowledged messages, ...
Read more >Monitoring RabbitMQ Performance With Datadog
The Datadog Agent checks your host for RabbitMQ performance metrics and sends them to Datadog. The Agent can also capture metrics and trace...
Read more >Collecting Metrics With RabbitMQ Monitoring Tools - Datadog
Tracing tools, Exchanges that report events from messages and application ... and exposes metrics about RabbitMQ objects such as queues, ...
Read more >Key Metrics for RabbitMQ Monitoring - Datadog
While RabbitMQ supports a number of protocols, it implements AMQP (Advanced Message Queuing Protocol) and extends some of its concepts.
Read more >Rabbitmq - Datadog
For tracking the performance of your messaging setup, check out these RabbitMQ monitoring tools. Monitoring RabbitMQ performance with Datadog. See your RabbitMQ ......
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
@tylerbenson our header value is null so the .toString() NPE
https://github.com/DataDog/dd-trace-java/blob/66b625c6a2191e616dd64cc8250271bcaaa44a58/dd-java-agent/instrumentation/rabbitmq-amqp-2.7/src/main/java/datadog/trace/instrumentation/rabbitmq/amqp/TextMapExtractAdapter.java#L15
@tylerbenson I did some preliminary local testing and the issue didn’t present itself with the snapshot. However, when I use the 0.17 agent, the issue also didn’t seem to be present either. I am still working on verifying that I can actually reproduce locally with our installed APM agent 0.17.