question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Zipkin trace headers contain multiple spans

See original GitHub issue

Describe the bug When enabling the tracing services in versions > 0.50 the Zipkin headers X-B3-Sampled, X-B3-Spanid, and X-B3-Traceid all appear to have multiple values concatenated into them.

Example responses from httpbin container:

On 0.40.2:

$ curl http://localhost:8080/httpbin/get
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Content-Length": "0",
    "Host": "localhost:8080",
    "User-Agent": "curl/7.54.0",
    "X-B3-Parentspanid": "5efc216a052d8f53",
    "X-B3-Sampled": "1",
    "X-B3-Spanid": "022c8f7c67c05af4",
    "X-B3-Traceid": "5efc216a052d8f53",
    "X-Envoy-Expected-Rq-Timeout-Ms": "3000",
    "X-Envoy-Original-Path": "/httpbin/get"
  },
  "origin": "172.21.0.3",
  "url": "http://localhost:8080/get"
}

On 0.51.2:

$ curl http://localhost:8080/httpbin/get
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Content-Length": "0",
    "Host": "localhost:8080",
    "User-Agent": "curl/7.54.0",
    "X-B3-Parentspanid": "31708df6d4f1b3a8",
    "X-B3-Sampled": "1,1",
    "X-B3-Spanid": "31708df6d4f1b3a8,ba3032afcb429cc9",
    "X-B3-Traceid": "31708df6d4f1b3a8,31708df6d4f1b3a8",
    "X-Envoy-Expected-Rq-Timeout-Ms": "3000",
    "X-Envoy-Internal": "true",
    "X-Envoy-Original-Path": "/httpbin/get"
  },
  "origin": "172.21.0.1",
  "url": "http://localhost:8080/get"
}

Trace JSON from zipkin from 0.51.2:

[
  {
    "traceId": "3d2e5cd9eadf6e66",
    "parentId": "3d2e5cd9eadf6e66",
    "id": "71f4e76b18114cbc",
    "kind": "CLIENT",
    "name": "router cluster_httpbin egress",
    "timestamp": 1553167127739170,
    "duration": 7395,
    "localEndpoint": {
      "serviceName": "ambassador-default",
      "ipv4": "172.21.0.4"
    },
    "tags": {
      "component": "proxy"
    }
  },
  {
    "traceId": "3d2e5cd9eadf6e66",
    "id": "3d2e5cd9eadf6e66",
    "kind": "CLIENT",
    "name": "localhost:8080",
    "timestamp": 1553167127738886,
    "duration": 7964,
    "localEndpoint": {
      "serviceName": "ambassador-default",
      "ipv4": "172.21.0.4"
    },
    "tags": {
      "component": "proxy",
      "downstream_cluster": "-",
      "guid:x-request-id": "e1891e18-ef0a-95a0-a85b-308bd5c67b62",
      "http.method": "GET",
      "http.protocol": "HTTP/1.1",
      "http.status_code": "200",
      "http.url": "http://localhost:8080/httpbin/get",
      "node_id": "test-id",
      "request_size": "0",
      "response_flags": "-",
      "response_size": "532",
      "upstream_cluster": "cluster_httpbin",
      "user_agent": "curl/7.54.0"
    }
  }
]

The X-B3-Spanid header contains the ids of both spans.

Confirmed the behavior is preset on the 0.50 releases and the 0.52 rc.

To Reproduce Can be reproduced in docker-compose setup as described here:

docker-compose.yaml:

version: '3.5'
services:
  ambassador:
    image: quay.io/datawire/ambassador:0.51.2
    ports:
    - 8080:80
    volumes:
    - ./config:/ambassador/ambassador-config
    environment:
    - AMBASSADOR_NO_KUBEWATCH=no_kubewatch
  httpbin:
    image: kennethreitz/httpbin
  zipkin:
    image: openzipkin/zipkin
    ports:
    - 9411:9411

config/ambassador.yaml:

---
apiVersion: ambassador/v1
kind: Module
name: ambassador
config: {}
---
apiVersion: ambassador/v1
kind: Mapping
name: http-bin
prefix: /httpbin/
service: httpbin
---
apiVersion: ambassador/v1
kind: TracingService
name: tracing
service: zipkin:9411
driver: zipkin

Then run:

$ docker-compose up -d
$ curl http://localhost:8080/httpbin/get

Expected behavior The X-B3-Spanid header should contain only the id of the “router cluster_httpbin egress” span, and the other X-B3-* headers should contain only a single value.

Versions (please complete the following information):

  • Ambassador: >= 0.50
  • Kubernetes environment: AWS EKS (Kubernetes 1.11) and docker-compose

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Reactions:2
  • Comments:9 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
gsagulacommented, Apr 5, 2019

Awesome, glad that it’s fixed.

0reactions
coryflucascommented, Apr 5, 2019

This appears to be fixed in 0.53.1 👍

Read more comments on GitHub >

github_iconTop Results From Across the Web

Zipkin trace headers contain multiple spans #1344 - GitHub
When enabling the tracing services in versions > 0.50 the Zipkin headers X-B3-Sampled , X-B3-Spanid , and X-B3-Traceid all appear to have ......
Read more >
Instrumenting a library · OpenZipkin
Zipkin merges spans together that share the same trace and span ID. The most common case of this is to merge a span...
Read more >
9. Customizations - Spring Cloud
Span ids are extracted from Zipkin-compatible (B3) headers (either Message or HTTP headers), to start or join an existing trace. Trace information is ......
Read more >
HTTP header manipulation - Envoy Proxy
The x-b3-parentspanid HTTP header is used by the Zipkin tracer in Envoy. The ParentSpanId is 64-bit in length and indicates the position of...
Read more >
Report Zipkin-format traces via Trace API
If you want to send more than one post, change the trace ID to a different value. Sending the same payload or span...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found