Trace headers not available in Tomcat access log

It seems that the Sleuth Trace/Spans are not available in the Tomcat access log. This only seems to be the case in the first initial application that receives a request. To give an example of this, I took the project from Spring Guides https://spring.io/guides/gs/routing-and-filtering / https://github.com/spring-guides/gs-routing-and-filtering. This project has a Netflix Zuul Gateway that sends requests to a Books application. In my fork of this project I use Spring boot 2.1.3.RELEASE and Spring Cloud Greenwich.SR1.

Access logs are configured for both applications with the following properties. A custom pattern is used to add the Sleuth Trace headers:

server.tomcat.accesslog.enabled=true
server.tomcat.accesslog.prefix=access
# set accesslog path
#server.tomcat.accesslog.directory=
server.tomcat.accesslog.pattern='{ "clientip":"%h", "log_date":"%t", "rawrequest_str":"%r", "response_int":%s, "bytes_int":%B, "response_time_int":%D, "referrer_stn":"%{Referer}i", "agent_stn":"%{User-agent}i", "X-B3-TraceId":"%{X-B3-TraceId}i", "X-B3-SpanId":"%{X-B3-SpanId}i", "X-B3-ParentSpanId":"%{X-B3-ParentSpanId}i" }'

Logs

Gateway access log

'{ "clientip":"0:0:0:0:0:0:0:1", "log_date":"[05/Apr/2019:19:10:16 +0200]", "rawrequest_str":"GET /books/available HTTP/1.1", "response_int":200, "bytes_int":27, "response_time_int":205, "referrer_stn":"-", "agent_stn":"insomnia/6.3.2", "X-B3-TraceId":"-", "X-B3-SpanId":"-", "X-B3-ParentSpanId":"-" }'
'{ "clientip":"0:0:0:0:0:0:0:1", "log_date":"[05/Apr/2019:19:10:17 +0200]", "rawrequest_str":"GET /books/available HTTP/1.1", "response_int":200, "bytes_int":27, "response_time_int":7, "referrer_stn":"-", "agent_stn":"insomnia/6.3.2", "X-B3-TraceId":"-", "X-B3-SpanId":"-", "X-B3-ParentSpanId":"-" }'
'{ "clientip":"0:0:0:0:0:0:0:1", "log_date":"[05/Apr/2019:19:10:18 +0200]", "rawrequest_str":"GET /books/available HTTP/1.1", "response_int":200, "bytes_int":27, "response_time_int":6, "referrer_stn":"-", "agent_stn":"insomnia/6.3.2", "X-B3-TraceId":"-", "X-B3-SpanId":"-", "X-B3-ParentSpanId":"-" }'
'{ "clientip":"0:0:0:0:0:0:0:1", "log_date":"[05/Apr/2019:19:10:18 +0200]", "rawrequest_str":"GET /books/available HTTP/1.1", "response_int":200, "bytes_int":27, "response_time_int":10, "referrer_stn":"-", "agent_stn":"insomnia/6.3.2", "X-B3-TraceId":"-", "X-B3-SpanId":"-", "X-B3-ParentSpanId":"-" }'

As you can see the trace headers are empty: "X-B3-TraceId":"-", "X-B3-SpanId":"-", "X-B3-ParentSpanId":"-" .

Simplefilter log (filter which basically logs the request in the gateway)

2019-04-05 19:10:16.623  INFO [-,e60907f3e137cccc,e60907f3e137cccc,false] 2146 --- [nio-8080-exec-1] hello.filters.pre.SimpleFilter           : GET request to http://localhost:8080/books/available
2019-04-05 19:10:17.368  INFO [-,64c924996acb4b7e,64c924996acb4b7e,false] 2146 --- [nio-8080-exec-2] hello.filters.pre.SimpleFilter           : GET request to http://localhost:8080/books/available
2019-04-05 19:10:18.083  INFO [-,f7a8250457d486c0,f7a8250457d486c0,false] 2146 --- [nio-8080-exec-3] hello.filters.pre.SimpleFilter           : GET request to http://localhost:8080/books/available
2019-04-05 19:10:18.840  INFO [-,06842a6aa982bb25,06842a6aa982bb25,false] 2146 --- [nio-8080-exec-4] hello.filters.pre.SimpleFilter           : GET request to http://localhost:8080/books/available

The trace headers are available here and I would expect the same headers in the gateway access log.

Book access log

'{ "clientip":"127.0.0.1", "log_date":"[05/Apr/2019:19:10:16 +0200]", "rawrequest_str":"GET /available HTTP/1.1", "response_int":200, "bytes_int":16, "response_time_int":67, "referrer_stn":"-", "agent_stn":"insomnia/6.3.2", "X-B3-TraceId":"e60907f3e137cccc", "X-B3-SpanId":"38e3977f2a64047c", "X-B3-ParentSpanId":"e60907f3e137cccc" }'
'{ "clientip":"127.0.0.1", "log_date":"[05/Apr/2019:19:10:17 +0200]", "rawrequest_str":"GET /available HTTP/1.1", "response_int":200, "bytes_int":16, "response_time_int":2, "referrer_stn":"-", "agent_stn":"insomnia/6.3.2", "X-B3-TraceId":"64c924996acb4b7e", "X-B3-SpanId":"1723e6468c175dcd", "X-B3-ParentSpanId":"64c924996acb4b7e" }'
'{ "clientip":"127.0.0.1", "log_date":"[05/Apr/2019:19:10:18 +0200]", "rawrequest_str":"GET /available HTTP/1.1", "response_int":200, "bytes_int":16, "response_time_int":2, "referrer_stn":"-", "agent_stn":"insomnia/6.3.2", "X-B3-TraceId":"f7a8250457d486c0", "X-B3-SpanId":"b5a3249ccf73b72e", "X-B3-ParentSpanId":"f7a8250457d486c0" }'
'{ "clientip":"127.0.0.1", "log_date":"[05/Apr/2019:19:10:18 +0200]", "rawrequest_str":"GET /available HTTP/1.1", "response_int":200, "bytes_int":16, "response_time_int":2, "referrer_stn":"-", "agent_stn":"insomnia/6.3.2", "X-B3-TraceId":"06842a6aa982bb25", "X-B3-SpanId":"899e2ffec895f751", "X-B3-ParentSpanId":"06842a6aa982bb25" }'

The trace headers are available in the Book’s access log, these are forwarded by the gateway. So it looks like when the incoming request contains the trace headers, these are also available in the access log. This is not the case with the gateway, as this is the first application that receives the request.

Could it be that at the moment an access log is created, no Trace has been created by Sleuth? There is also a similar issue https://github.com/spring-cloud/spring-cloud-sleuth/issues/1131 but with Netty access log.

Workaround

Now I have a workaround where the Trace headers are available in the first application that receives the request, based on the blog of inject-sleuth-header-on-first-request-with-tomcat. See branch tracing-accesslog-workaround of fork. A custom Tomcat valve (SleuthValve) is used to create a new span when the incoming request doesn’t have a TraceId.

class SleuthValve extends ValveBase {
  private final Tracer tracer;

  SleuthValve(Tracer tracer) {
    this.tracer = tracer;
  }

  @Override
  public void invoke(Request request, Response response) throws IOException, ServletException {
    enrichWithSleuthHeaderWhenMissing(tracer, request);

    Valve next = getNext();
    if (null == next) {
      // no next valve
      return;
    }
    next.invoke(request, response);
  }

  private static void enrichWithSleuthHeaderWhenMissing(final Tracer tracer, final Request request) {
    String header = request.getHeader("X-B3-TraceId");

    if (null == header) {
      org.apache.coyote.Request coyoteRequest = request.getCoyoteRequest();
      MimeHeaders mimeHeaders = coyoteRequest.getMimeHeaders();

      Span span = tracer.newTrace();

      addHeader(mimeHeaders, "X-B3-TraceId", span.context().traceIdString());
      addHeader(mimeHeaders, "X-B3-SpanId", span.context().traceIdString());
    }
  }

  private static void addHeader(MimeHeaders mimeHeaders, String traceIdName, String value) {
    MessageBytes messageBytes = mimeHeaders.addValue(traceIdName);
    messageBytes.setString(value);
  }
}

@Configuration
public class AccessLogSleuthConfiguration implements WebServerFactoryCustomizer<TomcatServletWebServerFactory> {

  private final SleuthValve sleuthValve;

  public AccessLogSleuthConfiguration(SleuthValve sleuthValve) {
    this.sleuthValve = sleuthValve;
  }

  @Override
  public void customize(TomcatServletWebServerFactory factory) {
    factory.addContextCustomizers(context -> context.getPipeline().addValve(sleuthValve));
  }

}

Logs

Gateway access log

'{ "clientip":"0:0:0:0:0:0:0:1", "log_date":"[05/Apr/2019:19:18:20 +0200]", "rawrequest_str":"GET /books/available HTTP/1.1", "response_int":200, "bytes_int":27, "response_time_int":186, "referrer_stn":"-", "agent_stn":"insomnia/6.3.2", "X-B3-TraceId":"c70efc5d25b70730", "X-B3-SpanId":"c70efc5d25b70730", "X-B3-ParentSpanId":"-" }'
'{ "clientip":"0:0:0:0:0:0:0:1", "log_date":"[05/Apr/2019:19:18:21 +0200]", "rawrequest_str":"GET /books/available HTTP/1.1", "response_int":200, "bytes_int":27, "response_time_int":7, "referrer_stn":"-", "agent_stn":"insomnia/6.3.2", "X-B3-TraceId":"c337d4c36da9e186", "X-B3-SpanId":"c337d4c36da9e186", "X-B3-ParentSpanId":"-" }'
'{ "clientip":"0:0:0:0:0:0:0:1", "log_date":"[05/Apr/2019:19:18:22 +0200]", "rawrequest_str":"GET /books/available HTTP/1.1", "response_int":200, "bytes_int":27, "response_time_int":6, "referrer_stn":"-", "agent_stn":"insomnia/6.3.2", "X-B3-TraceId":"9b441be7bdf1559e", "X-B3-SpanId":"9b441be7bdf1559e", "X-B3-ParentSpanId":"-" }'
'{ "clientip":"0:0:0:0:0:0:0:1", "log_date":"[05/Apr/2019:19:18:23 +0200]", "rawrequest_str":"GET /books/available HTTP/1.1", "response_int":200, "bytes_int":27, "response_time_int":5, "referrer_stn":"-", "agent_stn":"insomnia/6.3.2", "X-B3-TraceId":"f079246af5fbba84", "X-B3-SpanId":"f079246af5fbba84", "X-B3-ParentSpanId":"-" }'

SimpleFilter log

2019-04-05 19:18:20.546  INFO [-,c70efc5d25b70730,c70efc5d25b70730,false] 2526 --- [nio-8080-exec-1] hello.filters.pre.SimpleFilter           : GET request to http://localhost:8080/books/available
2019-04-05 19:18:21.296  INFO [-,c337d4c36da9e186,c337d4c36da9e186,false] 2526 --- [nio-8080-exec-2] hello.filters.pre.SimpleFilter           : GET request to http://localhost:8080/books/available
2019-04-05 19:18:21.997  INFO [-,9b441be7bdf1559e,9b441be7bdf1559e,false] 2526 --- [nio-8080-exec-3] hello.filters.pre.SimpleFilter           : GET request to http://localhost:8080/books/available
2019-04-05 19:18:23.586  INFO [-,f079246af5fbba84,f079246af5fbba84,false] 2526 --- [nio-8080-exec-4] hello.filters.pre.SimpleFilter           : GET request to http://localhost:8080/books/available

Book access log

'{ "clientip":"127.0.0.1", "log_date":"[05/Apr/2019:19:18:20 +0200]", "rawrequest_str":"GET /available HTTP/1.1", "response_int":200, "bytes_int":16, "response_time_int":59, "referrer_stn":"-", "agent_stn":"insomnia/6.3.2", "X-B3-TraceId":"c70efc5d25b70730", "X-B3-SpanId":"ff0ff504dbf88036", "X-B3-ParentSpanId":"c70efc5d25b70730" }'
'{ "clientip":"127.0.0.1", "log_date":"[05/Apr/2019:19:18:21 +0200]", "rawrequest_str":"GET /available HTTP/1.1", "response_int":200, "bytes_int":16, "response_time_int":2, "referrer_stn":"-", "agent_stn":"insomnia/6.3.2", "X-B3-TraceId":"c337d4c36da9e186", "X-B3-SpanId":"c9942540270d5b92", "X-B3-ParentSpanId":"c337d4c36da9e186" }'
'{ "clientip":"127.0.0.1", "log_date":"[05/Apr/2019:19:18:22 +0200]", "rawrequest_str":"GET /available HTTP/1.1", "response_int":200, "bytes_int":16, "response_time_int":2, "referrer_stn":"-", "agent_stn":"insomnia/6.3.2", "X-B3-TraceId":"9b441be7bdf1559e", "X-B3-SpanId":"12cd013e5ad7bc0d", "X-B3-ParentSpanId":"9b441be7bdf1559e" }'
'{ "clientip":"127.0.0.1", "log_date":"[05/Apr/2019:19:18:23 +0200]", "rawrequest_str":"GET /available HTTP/1.1", "response_int":200, "bytes_int":16, "response_time_int":2, "referrer_stn":"-", "agent_stn":"insomnia/6.3.2", "X-B3-TraceId":"f079246af5fbba84", "X-B3-SpanId":"df0cddef94547799", "X-B3-ParentSpanId":"f079246af5fbba84" }'

Is this a good solution to make the trace headers available in the access log? It would be nice if Sleuth can take care of this.

[marcingrzejszczak]

I’m so sorry that it took me that long. Please check out 3.1.0-SNAPSHOT version https://github.com/spring-cloud/spring-cloud-sleuth/commit/2dd82624e597d1dd39104fcd428701dc6d672be6

[madjaye]

It is not a problem in that the SleuthValve workaround described by @thanus above works (with some tweaks to make it work with latest dependency versions). But the original “problem” does still exist…