Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

linkerd fails to route correctly for a recreated k8s namespace

See original GitHub issue

Description

When linkerd is set up to route to Kubernetes services, it seems to cache something related to the service such that if I delete and recreate the namespace, it can no longer route to services in that namespace.

No hosts are available for /http/1.1/GET/nginx, Dtab.base=[/k8s=>/#/io.l5d.k8s;/ns=>/k8s/master;/ns=>/k8s/production;/srv=>/ns/http;/http/*/*=>/srv], Dtab.local=[/ns=>/k8s/temp]. Remote Info: Not Available

The same thing seems to happen if I delete everything in the namespace and redeploy the services.

Environment

Linkerd configuration (running as DaemonSet): https://gist.github.com/agunnerson-ibm/7884bf73964e12af93c45ad4ca1d758b#file-l5d-config-yaml
Kubernetes 1.5.3 infrastructure and kubelet 1.5.3 sidecar container
Kubernetes service and deployment: https://gist.github.com/agunnerson-ibm/7884bf73964e12af93c45ad4ca1d758b#file-nginx-yaml

Reproduction steps

I’m assuming that linkerd is set up similar to the configuration above.

Given the dtab in the linkerd config:

/k8s      => /#/io.l5d.k8s;
/ns       => /k8s/master;
/ns       => /k8s/production;
/srv      => /ns/http;
/http/*/* => /srv;

Deploy nginx.yaml to temp namespace:

kubectl -n temp apply -f nginx.yaml

Hit the service to make sure it works:

http_proxy=http://kube-node-01:4140 curl -v http://nginx -H 'l5d-dtab: /ns => /k8s/temp'

Delete the namespace and recreate it:

kubectl delete ns temp
# Wait for deletion to complete (see `kubectl get ns`)
kubectl create ns temp

Redeploy nginx.yaml to recreated temp namespace:

kubectl -n temp apply -f nginx.yaml

Linkerd can no longer route to the temp namespace:

$ http_proxy=http://kube-node-01:4140 curl http://nginx -H 'l5d-dtab: /ns => /k8s/temp'
No hosts are available for /http/1.1/GET/nginx, Dtab.base=[/k8s=>/#/io.l5d.k8s;/ns=>/k8s/master;/ns=>/k8s/production;/srv=>/ns/http;/http/*/*=>/srv], Dtab.local=[/ns=>/k8s/temp]. Remote Info: Not Available

Issue Analytics

State:
Created 6 years ago
Comments:10 (5 by maintainers)

Top GitHub Comments

2reactions

agunnerson-ibmcommented, Oct 3, 2017

It looks like this bug was finally fixed in linkerd 1.2.1! I was able reproduce it in 1.2.0, but not 1.2.1.

0reactions

wmorgancommented, Oct 3, 2017

@agunnerson-ibm awesome, glad to hear it!

Top Results From Across the Web

Troubleshooting

This error indicates that the proxies running in the Linkerd control plane are not healthy. Ensure that Linkerd has been installed with all...

Adding your services to Linkerd

Meshing a Kubernetes resource is typically done by annotating the resource (or its namespace) with the linkerd.io/inject: enabled Kubernetes annotation.

K8s: Linkerd fails to resolve service - Kubernetes

There is a known issue where linkerd/namerd will stop getting updates from a k8s namespace if the namespace is deleted and recreated.

App is unable to route through linkerd in k8s - Kubernetes

App is unable to route through linkerd in k8s ; Setup: I have an app which routes to given endpoints, I get the...

Upgrading Linkerd

Since pods are immutable in Kubernetes, Linkerd is unable to simply update ... above will result in the recreation of the linkerd-multicluster namespace, ......