Brave JMS instrumentation does not work properly when ActiveMQ BytesMessage is received
See original GitHub issueDescription
When org.apache.activemq.command.ActiveMQBytesMessage JMS message is received, and sleuth is enabled, the traceId and spanId are working as expected, but the content (or payload) of the message is lost. This occurs when using spring boot 2.5.7 and spring cloud 2020.0.4 (brave 5.13.2, and ActiveMQ Client 5.16.3)! I came to the conclusion that ActiveMQBytesMessage does not behave correctly after brave instrumentation JMS edits the message to remove some headers. However, I’m not sure if this is an issue on the ActiveMQ side, or Brave instrumentation does not edit the BytesMessage correctly.
Steps to Reproduce
Before calling brave.jms.PropertyFilter#filterProperties(javax.jms.Message, java.util.Set<java.lang.String>, java.util.List<java.lang.Object>)
method we can see the following in the BytesMessage:
After calling brave.jms.PropertyFilter#filterProperties(javax.jms.Message, java.util.Set<java.lang.String>, java.util.List<java.lang.Object>)
method we can see the following in the BytesMessage:
To be more precicelly, after calling org.apache.activemq.command.ActiveMQBytesMessage#setObjectProperty
the content is moved in dataOut/bytesOut.
Next, if you try to read something from BytesMessage, using org.apache.activemq.command.ActiveMQBytesMessage#readBytes
method for example, then you will discover that message has no content, and this is unexpected:
Expected Behaviour
After setting the properties back, via org.apache.activemq.command.ActiveMQBytesMessage#setObjectProperty
, the BytesMessage content should be restored, before reaching the MessageListener or MessageConverter.
For example, after calling brave.jms.PropertyFilter#filterProperties(javax.jms.Message, java.util.Set<java.lang.String>, java.util.List<java.lang.Object>)
method, you can reset the BytesMessage using reset()
method:
Note that content was restored, and dataIn, dataOut, and bytesOut were cleared.
After that you will be able to read the content:
Things to consider
Have a look at the org.apache.activemq.command.ActiveMQBytesMessage#setObjectProperty
method:
The initializeWritingNoCheck() method will simply move the content
to dataOut
/bytesOut
. As a result, content
will be null
and dataOut
/bytesOut
will contain the bytes moved from content
. Note dataIn
remains null
.
Have a look at org.apache.activemq.command.ActiveMQBytesMessage#read*
methods, for example:
They call initializeReading()
method, that does the following:
I.e. when content
is missing, the dataIn
is initialized with an empty DataInputStream. In my opinion here is the issue ^, meaning this is a bug in ActiveMQ client. But I’m not sure about this, that’s why I created this bug here.
Note: I do not know who should do the fix, that’s why I have created this bug, and additionally I have created a bug on the ActiveMQ client side, in hope to obtain a resolution ASAP… check https://issues.apache.org/jira/browse/AMQ-8417
Issue Analytics
- State:
- Created 2 years ago
- Reactions:2
- Comments:6
Top GitHub Comments
I also got bitten by this error after upgrading from activemq 5.15.15 to 5.16.4, And brave-instrumentation-jms from 512.7 to 5.13.7 (via spring-cloud-dependencies). Our workaround is to call
message.reset()
in our own message receiver.It just happened to us. We had some @JmsListener methods that were working fine and simply adding spring-cloud-starter-sleuth and some brave dependencies, the listeners started failing due to JAXB throwing End Of File exceptions. We suspected it was because the tracing was not resetting the XML content after doing something with it, and this post seems to confirm it.
Removing the dependencies made all work again, without any other change.
In the end, we avoided the problem altogether with
spring.sleuth.messaging.jms.enabled=false
as the tracing, in our case, was included for HttpClient calls, not JMS.Edit: Note that we cannot call the reset() method as the problems happens when Spring tries to unmarshall the message to create the bean/DTO to pass to our code, so the problem occurs before our code is called.