Document how log injection works

We are using logback and slf4j, plus logstash to output JSON in our logs. Lately I was working on adding trace id injection.

Reading docs here: https://docs.datadoghq.com/tracing/connect_logs_and_traces/java/?tab=slf4jandlogback it looked like it would be very simple, however in practice this turned out to be much more complicated and required reading sources of this library. I believe the reason is lack of documentation on subject.

First of all “Starting in version 0.74.0, the Java tracer automatically injects trace correlation identifiers into logs” is not really true, because we still have to write code which would take trace and span id from MDC and put it to json output. Mostly because we (and probably a lot of other projects) already have some customizations there, so whatever agent is doing that is supposed to work automatically - did not really worked.

Then I looked at “Manual injection” paragraph and figured that I could just make my provider taking trace from MDC and placing to JSON. This looked like this (in Kotlin):

class DatadogProvider : AbstractFieldJsonProvider<ILoggingEvent>() {

    override fun writeTo(generator: JsonGenerator, event: ILoggingEvent) {
        val traceId = MDC.get("dd.trace_id")
        traceId?.let {
            generator.writeStringField("trace_id", it)
        }
        val spanId = MDC.get("dd.span_id")
        spanId?.let {
            generator.writeStringField("span_id", it)
        }
    }

}

And did in fact worked up to version of agent 0.76.0 , where this change: https://github.com/DataDog/dd-trace-java/pull/2440 broke this approach.

Interestingly, changelog does not even list this as a breaking change (which it of course is). It is listed as “Log injection enabled by default”, while PR itself is named “Log injection rework”, which is a very big difference.

Reading the code of the change I figured that I need to refer to getMDCPropertyMap from ILoggingEvent rather than to MDC.

Now I had to rewrite my provider like this:

class DatadogProvider : AbstractFieldJsonProvider<ILoggingEvent>() {

    override fun writeTo(generator: JsonGenerator, event: ILoggingEvent) {
        // It is important to use event.mdcPropertyMap, which is instrumented by
        // Datadog java agent, rather than MDC.get, which is not
        val traceId = event.mdcPropertyMap["dd.trace_id"]
        traceId?.let {
            generator.writeStringField("trace_id", it)
        }
        val spanId = event.mdcPropertyMap["dd.span_id"]
        spanId?.let {
            generator.writeStringField("span_id", it)
        }
    }

}

, which appears to be working with last version.

I believe that documentation lacks any details on how this works and it must be expanded a lot.

[strowk]

Thanks for examples @randomanderson

With Slf4j + logback + logstash encoder you should not need to write your own provider. By default, the MdcJsonProvider, writes all fields in the event’s MDC to the generated json

Is there a documentation explaining how I can use MdcJsonProvider to configure datadog log injection? We list all used providers explicitely in logback.xml currently, I had no idea that using that one is the right way to go, that is why had to write my own. If this would’ve been documented somewhere, that’d be great!

[arlyon]

I have also spent the last 6 or so jumping between bad documentation only to stumble on this. For the record, here is my xml since I’m not sure how to use MdcJsonProvider. Weirdly, service, and and version are being injected automatically but these arent…

<configuration>
  <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
    <encoder class="net.logstash.logback.encoder.LogstashEncoder">
      <includeMdcKeyName>dd.trace_id</includeMdcKeyName>
      <includeMdcKeyName>dd.span_id</includeMdcKeyName>
    </encoder>
  </appender>
  <root level="INFO">
    <appender-ref ref="CONSOLE"/>
  </root>
</configuration>