Istio and zipkin communication to jaeger

I have problem with istio and jaeger communication.

I have simple springboot app. It sends spans to jaeger server (everything wihtin kubernetes cluster). Everything works fine without istio.

Spans are sampled and sent to jaeger.

        ...
        zipkin:
            baseUrl: http://zipkinjaeger.default.svc.cluster.local:9411
            sender:
                type: web
        sleuth:
            sampler:
                percentage: 1
                probability: 1
       ...

The same app with the same configuration stops sending spans when I setup istio in my environment. There is no errors in logs.

I just see that span is never sampled.

Spring boot app with istio injected in namespace apps zipkin (jaeger) server without istio injected in default domain.

In Istio ingress gateway I see only warning: I have no idea whether it is source of my problem.

[2019-02-20 20:36:57.134][21][warning][config] bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:70] gRPC config for type.googleapis.com/envoy.api.v2.ClusterLoadAssignment rejected: malformed IP address: zipkinjaeger.default.svc.cluster.local. Consider setting resolver_name or setting cluster type to 'STRICT_DNS' or 'LOGICAL_DNS'
[

Service entry object created:

---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: zipkinjaeger.default.svc.cluster.local
spec:
  hosts:
  - zipkinjaeger.default.svc.cluster.local
  ports:
  - number: 9411
    name: tcp
    protocol: TCP
  resolution: DNS
  location: MESH_INTERNAL
---

Environment:

Spring boot 2.1.3 spring-cloud.version Greenwich.RELEASE

4 node kubernetes cluster ubuntu 18:

 ./istioctl version
Version: 1.0.6
GitRevision: 98598f88f6ee9c1e6b3f03b652d8e0e3cd114fa2
User: root@464fc845-2bf8-11e9-b805-0a580a2c0506
Hub: docker.io/istio
GolangVersion: go1.10.4
BuildStatus: Clean

root@k8smaster:~/istio-1.0.6/bin#
root@k8smaster:~/istio-1.0.6/bin#
root@k8smaster:~/istio-1.0.6/bin#
root@k8smaster:~/istio-1.0.6/bin#
root@k8smaster:~/istio-1.0.6/bin# kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:08:12Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:00:57Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
root@k8smaster:~/istio-1.0.6/bin#

I’m almost sure it is istio but I have no idea how I can troubleshoot it. Is there property I can set to check why spans are not sampled?

[shakuzen]

with istio I have:

2019-02-21 10:05:27.326 INFO [SpringBootWeb,b2224f8059cabaff,b2224f8059cabaff,false] 16024 --- [nio-7070-exec-1] mk.WebController : Message sent

The application and it’s configuration is the same.

I’m almost sure istio blocks something.

I would guess Istio is sending a sampled header, to control the sampling. Sleuth is just respecting the sampling decision it was sent. See https://istio.io/docs/tasks/telemetry/distributed-tracing/#trace-sampling

In other configurations, Istio defaults to generating trace spans for 1 out of every 100 requests (sampling rate of of 1%).

[marcin-kasinski]

Thank you. I think you are right. I will check it.