Distributed Tracing with APM server; with Python

I’ve been doing POC tests on different Tracing technologies (Jaeger, Zipkin, Stackdriver trace, Istio (still Jaeger or Zipkin. Different concepts though)), and now I am with Elasticsearch APM module, and I see it is more or less the same concept. You start counting a trace when you start a request, and end it when you get a response.

I’ve generated some traces, and am able to see them on Kibana, but I see the traces separately, which makes sense, as every time; in each service I initialize a new Tracer object, and this gets a new ID.

Now, when I want to see the cascade view of several service spans, or several spans of the same service, I pass the Trace ID to the next service, so this one will initialize the tracer with this ID, and will generate a new Span ID, and attach to the same trace.

I’ve been reading the docs, for Python, and the only method that suits this is elasticapm.set_context(), but everything I found in the docs is this:

def set_context(data, key=“custom”): “”" Attach contextual data to the current transaction and errors that happen during the current transaction. If the transaction is not sampled, this function becomes a no-op. :param data: a dictionary, or a callable that returns a dictionary :param key: the namespace for this data “”" …

I would like to know if this is the right way of doing this, or I am completely off the track.

[basepi]

Alright, I see the disconnect. I have a working example, modified from your example above, in this gist

The problem was that you didn’t quite have the instrumentation for service 2a/b and 3 correct. When you were looking at the trace, all you were seeing were the transaction and two spans (for the two network calls) from service 1.

This is because our instrumentation for flask requires us to connect to flask’s signals, which we only do if you set up our flask integration, as documented here.

Otherwise the flask routing doesn’t get instrumented, which means that while our headers are there from service 1, the agent doesn’t know to look for them. In order to create a transaction that actually uses incoming http headers, you have to use begin_transaction with a TraceParent object, like we do here (in the flask integration code). So, if you ever needed to do distributed tracing in an unsupported framework, you’d do something like that.

Luckily, if you use our official integrations, we do all that hard work for you!

This is the waterfall I see when I run the example in my gist:

Much better!

Please keep me posted if anything I explained wasn’t clear. We’re here to help!

[beniwohli]

It looks like all questions have been addressed. I’ll close this for now 😃