JDBC queries do not get instrumented when using OpenTracing Bridge

Describe the bug We are using Apache Camel with their OpenTracing component and the Elastic APM agent with the OpenTracing plugin. We want to trace our camel routes and also like to see the executed jdbc queries. The camel routes are reported as expected producing traces, but we do not see the jdbc queries called by the routes. Even adding @CaptureSpan on the jdbc query methods has no effect, as the agent rejects the span because he cannot find an open transaction.

To Reproduce I have created a simple demo app to reproduce the behavior: https://github.com/pnowak85/camel-apm-demo

Replace the db properties in the application.properties and execute the app.

The app has one timer route, calling a direct route which then executes a jdbc query. There is also a little spring scheduled service executing another jdbc query to ensure that the jdbc instrumentation is working as expected.

The agent is configured as followed:

java -javaagent:path/elastic-apm-agent-1.8.0.jar -Delastic.apm.service_name=camel-apm-demo -Delastic.apm.server_url=http://localhost:8200 -Delastic.apm.log_level=DEBUG -Delastic.apm.application_packages=space.pezi -Delastic.apm.trace_methods=space.pezi -jar .\camel-apm-demo-0.0.1-SNAPSHOT.jar               

Expected behavior A trace starting from the timer route, including the querydb route and the executed jdbc query.

Debug logs

Click to expand

2019-08-06 11:17:29.967 [Camel (camel-1) thread #1   - timer://demotimer] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Type match for instrumentation CreateSpanInstrumentation: name(equals(co.elastic.apm.opentracing.ApmSpanBuilder)) matches class co.elastic.apm.opentracing.ApmSpanBuilder
2019-08-06 11:17:29.973 [Camel (camel-1) thread #1   - timer://demotimer] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Method match for instrumentation CreateSpanInstrumentation: name(equals(createSpan)) matches private java.lang.Object co.elastic.apm.opentracing.ApmSpanBuilder.createSpan(java.lang.Object,java.lang.Iterable)
2019-08-06 11:17:29.983 [Camel (camel-1) thread #1   - timer://demotimer] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - increment references to '' 00-ee02357b902f2a5339bd317fd827b29f-ac81dcb477863716-01 (75db19c7) (1)
2019-08-06 11:17:29.983 [Camel (camel-1) thread #1   - timer://demotimer] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - startTransaction '' 00-ee02357b902f2a5339bd317fd827b29f-ac81dcb477863716-01 (75db19c7) {
2019-08-06 11:17:29.985 [Camel (camel-1) thread #1   - timer://demotimer] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Type match for instrumentation SpanContextInstrumentation: name(equals(co.elastic.apm.opentracing.TraceContextSpanContext)) matches class co.elastic.apm.opentracing.TraceContextSpanContext
2019-08-06 11:17:29.987 [Camel (camel-1) thread #1   - timer://demotimer] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Method match for instrumentation SpanContextInstrumentation: name(equals(baggageItems)) matches public java.lang.Iterable co.elastic.apm.opentracing.TraceContextSpanContext.baggageItems()
2019-08-06 11:17:29.999  INFO 40776 --- [mer://demotimer] route1                                   : Running
2019-08-06 11:17:30.003 [Camel (camel-1) thread #1   - timer://demotimer] DEBUG co.elastic.apm.agent.impl.transaction.Span - startSpan '' 00-ee02357b902f2a5339bd317fd827b29f-10621886dae00f51-01 (6025d54f) {
2019-08-06 11:17:30.003 [Camel (camel-1) thread #1   - timer://demotimer] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - increment references to '' 00-ee02357b902f2a5339bd317fd827b29f-10621886dae00f51-01 (6025d54f) (1)
2019-08-06 11:17:30.006 [Camel (camel-1) thread #1   - timer://demotimer] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - increment references to '' 00-ee02357b902f2a5339bd317fd827b29f-29eabd33ce6cb7ad-01 (488f08de) (1)
2019-08-06 11:17:30.006 [Camel (camel-1) thread #1   - timer://demotimer] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - startTransaction '' 00-ee02357b902f2a5339bd317fd827b29f-29eabd33ce6cb7ad-01 (488f08de) {
2019-08-06 11:17:30.008 [Camel (camel-1) thread #1   - timer://demotimer] DEBUG co.elastic.apm.agent.plugin.api.CaptureSpanInstrumentation - Not creating span for DemoBean#queryTable1 because there is no currently active span.
Camel routebean DB query executed - 18798 rows
2019-08-06 11:17:30.018 [Camel (camel-1) thread #1   - timer://demotimer] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - } endTransaction 'querydb' 00-ee02357b902f2a5339bd317fd827b29f-29eabd33ce6cb7ad-01 (488f08de)
2019-08-06 11:17:30.019 [Camel (camel-1) thread #1   - timer://demotimer] DEBUG co.elastic.apm.agent.impl.transaction.Span - } endSpan 'querydb' 00-ee02357b902f2a5339bd317fd827b29f-10621886dae00f51-01 (6025d54f)
2019-08-06 11:17:30.020 [Camel (camel-1) thread elastic/apm-agent-java#1 - timer://demotimer] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - } endTransaction 'demotimer' 00-ee02357b902f2a5339bd317fd827b29f-ac81dcb477863716-01 (75db19c7)
2019-08-06 11:17:30.022 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving TRANSACTION event (sequence 2)
2019-08-06 11:17:30.022 [apm-reporter] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'querydb' 00-ee02357b902f2a5339bd317fd827b29f-29eabd33ce6cb7ad-01 (488f08de) (0)
2019-08-06 11:17:30.023 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving SPAN event (sequence 3)
2019-08-06 11:17:30.024 [apm-reporter] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'querydb' 00-ee02357b902f2a5339bd317fd827b29f-10621886dae00f51-01 (6025d54f) (0)
2019-08-06 11:17:30.025 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving TRANSACTION event (sequence 4)
2019-08-06 11:17:30.025 [apm-reporter] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'demotimer' 00-ee02357b902f2a5339bd317fd827b29f-ac81dcb477863716-01 (75db19c7) (0)
2019-08-06 11:17:38.401 [pool-1-thread-1] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - increment references to '' 00-645fe461f16957b0456cff179173cd40-5d47dc7f180a0795-01 (488f08de) (1)
2019-08-06 11:17:38.403 [pool-1-thread-1] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - startTransaction '' 00-645fe461f16957b0456cff179173cd40-5d47dc7f180a0795-01 (488f08de) {
2019-08-06 11:17:38.404 [apm-request-timeout-timer] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Request flush because the request timeout occurred

Actual output

As you can see, the jdbc query is not shown, but it is working if you look at the background thread execution:

Is this a bug in the Elastic OpenTracing Bridge or is are the spans/transactions not correctly created on the camel opentracing side?

[pnowak85]

Thanks for your help, i will try activating the spans in the ActiveSpanManager.

I think the ticket can be closed, as seems to be a camel-opentracing issue.

[eyalkoren]

In which location does the agent look for active spans?

The agent always looks for an active span when creating a new span, otherwise it will be an orphan that we can’t relate to anything and it will be dropped.

Shouldn’t the OpenTracingTracer start a span here and store it via the ElasticApmTracer/ApmScopeManager?

It seems that the Camel OpenTracing tracer doesn’t use the OpenTracing activate and startActive APIs. Instead, it uses something specific for the Camel Exchange (see the ActiveSpanManager code). Therefore, our agent is not informed of these activations and deactivations. You can hook to the ActiveSpanManager#getSpan API and activate/deactivate the returned span manually. However, for that you need to have the Camel Exchange reference and a dependency in the camel-opentracing lib. If you do that- you don’t need to add Elastic spans created through APIs (eg @CpatureSpan).