Zipkin trace headers contain multiple spans
See original GitHub issueDescribe the bug
When enabling the tracing services in versions > 0.50 the Zipkin headers X-B3-Sampled
, X-B3-Spanid
, and X-B3-Traceid
all appear to have multiple values concatenated into them.
Example responses from httpbin container:
On 0.40.2:
$ curl http://localhost:8080/httpbin/get
{
"args": {},
"headers": {
"Accept": "*/*",
"Content-Length": "0",
"Host": "localhost:8080",
"User-Agent": "curl/7.54.0",
"X-B3-Parentspanid": "5efc216a052d8f53",
"X-B3-Sampled": "1",
"X-B3-Spanid": "022c8f7c67c05af4",
"X-B3-Traceid": "5efc216a052d8f53",
"X-Envoy-Expected-Rq-Timeout-Ms": "3000",
"X-Envoy-Original-Path": "/httpbin/get"
},
"origin": "172.21.0.3",
"url": "http://localhost:8080/get"
}
On 0.51.2:
$ curl http://localhost:8080/httpbin/get
{
"args": {},
"headers": {
"Accept": "*/*",
"Content-Length": "0",
"Host": "localhost:8080",
"User-Agent": "curl/7.54.0",
"X-B3-Parentspanid": "31708df6d4f1b3a8",
"X-B3-Sampled": "1,1",
"X-B3-Spanid": "31708df6d4f1b3a8,ba3032afcb429cc9",
"X-B3-Traceid": "31708df6d4f1b3a8,31708df6d4f1b3a8",
"X-Envoy-Expected-Rq-Timeout-Ms": "3000",
"X-Envoy-Internal": "true",
"X-Envoy-Original-Path": "/httpbin/get"
},
"origin": "172.21.0.1",
"url": "http://localhost:8080/get"
}
Trace JSON from zipkin from 0.51.2:
[
{
"traceId": "3d2e5cd9eadf6e66",
"parentId": "3d2e5cd9eadf6e66",
"id": "71f4e76b18114cbc",
"kind": "CLIENT",
"name": "router cluster_httpbin egress",
"timestamp": 1553167127739170,
"duration": 7395,
"localEndpoint": {
"serviceName": "ambassador-default",
"ipv4": "172.21.0.4"
},
"tags": {
"component": "proxy"
}
},
{
"traceId": "3d2e5cd9eadf6e66",
"id": "3d2e5cd9eadf6e66",
"kind": "CLIENT",
"name": "localhost:8080",
"timestamp": 1553167127738886,
"duration": 7964,
"localEndpoint": {
"serviceName": "ambassador-default",
"ipv4": "172.21.0.4"
},
"tags": {
"component": "proxy",
"downstream_cluster": "-",
"guid:x-request-id": "e1891e18-ef0a-95a0-a85b-308bd5c67b62",
"http.method": "GET",
"http.protocol": "HTTP/1.1",
"http.status_code": "200",
"http.url": "http://localhost:8080/httpbin/get",
"node_id": "test-id",
"request_size": "0",
"response_flags": "-",
"response_size": "532",
"upstream_cluster": "cluster_httpbin",
"user_agent": "curl/7.54.0"
}
}
]
The X-B3-Spanid
header contains the ids of both spans.
Confirmed the behavior is preset on the 0.50 releases and the 0.52 rc.
To Reproduce Can be reproduced in docker-compose setup as described here:
docker-compose.yaml:
version: '3.5'
services:
ambassador:
image: quay.io/datawire/ambassador:0.51.2
ports:
- 8080:80
volumes:
- ./config:/ambassador/ambassador-config
environment:
- AMBASSADOR_NO_KUBEWATCH=no_kubewatch
httpbin:
image: kennethreitz/httpbin
zipkin:
image: openzipkin/zipkin
ports:
- 9411:9411
config/ambassador.yaml:
---
apiVersion: ambassador/v1
kind: Module
name: ambassador
config: {}
---
apiVersion: ambassador/v1
kind: Mapping
name: http-bin
prefix: /httpbin/
service: httpbin
---
apiVersion: ambassador/v1
kind: TracingService
name: tracing
service: zipkin:9411
driver: zipkin
Then run:
$ docker-compose up -d
$ curl http://localhost:8080/httpbin/get
Expected behavior
The X-B3-Spanid
header should contain only the id of the “router cluster_httpbin egress” span, and the other X-B3-*
headers should contain only a single value.
Versions (please complete the following information):
- Ambassador: >= 0.50
- Kubernetes environment: AWS EKS (Kubernetes 1.11) and docker-compose
Issue Analytics
- State:
- Created 5 years ago
- Reactions:2
- Comments:9 (5 by maintainers)
Awesome, glad that it’s fixed.
This appears to be fixed in 0.53.1 👍