l5d incorrectly reporting/caching endpoints after service is scaled to 0

Issue Type:

• Bug report
• Feature request

What happened: l5d incorrectly reports the number endpoints available after a service has been scaled all the way down to 0 and stays in an invalid state even after scaling up.

Linkerd logs reports service failure and does not correctly remove the stale node IPs:

|l5d-h5lwn::l5d              | E 0911 23:08:35.537 UTC THREAD28 TraceId:3656019fcecb838b: service failure: Failure(connection timed out: /10.44.5.204:9090 at remote address: /10.44.5.204:9090. Remote Info: Not Available, flags=0x09) with RemoteInfo -> Upstream Address: /10.44.2.192:38684, Upstream Client Id: Not Available, Downstream Address: /10.44.5.204:9090, Downstream Client Id: %/io.l5d.k8s.localnode/10.44.5.200/#/io.l5d.k8s.ns/http/reservation-srv, Trace Id: 3656019fcecb838b.be266a254a63552e<:b60c58529334b85c
|l5d-2kdvb::l5d              | E 0911 23:08:35.536 UTC THREAD50 TraceId:d1893092c9eeb8e1: service failure: Failure(connection timed out: /10.44.2.190:9090 at remote address: /10.44.2.190:9090. Remote Info: Not Available, flags=0x09) with RemoteInfo -> Upstream Address: /10.44.2.192:36334, Upstream Client Id: Not Available, Downstream Address: /10.44.2.190:9090, Downstream Client Id: %/io.l5d.k8s.localnode/10.44.2.192/#/io.l5d.k8s.ns/http/reservation-srv, Trace Id: d1893092c9eeb8e1.664bb4085c05d9f8<:7bdef9c6b046c20e

What you expected to happen: l5d should correctly report new endpoints as available for a service as replica count goes up an down regardless of the number of replicas.

How to reproduce it (as minimally and precisely as possible):

1. Scale down a service replica to 0 (at this point, service available is already incorrect).
2. Scale service back up (still incorrect).

Anything else we need to know?:

• Running l5d as a daemonset.
• dtab is correctly reporting the set of prod IPs but requests themselves are failing to resolve.

Environment:

• linkerd/namerd version, config files: 1.2.0 config
• Platform, version, and config files (Kubernetes, DC/OS, etc): Kubernetes
• Cloud provider or hardware configuration: GKE

[jsurdilla]

That’s amazing, looking forward to checking it out. Thank you!

[hawkw]

Hi @jsurdilla and @Taik, pull request #1638 should fix this issue. We’ll try and get the fix out as soon as possible!