Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[0.9.1] Failure Accrual and FD Exhaustion

See original GitHub issue

I’ve been seeing a slow FD leak that correlates very closely w/ failureaccrual.

The metric that increases forever is jvm/fd_count in my case we are exhausting 16k fds.

A few instances of failureaccrual seem to be consuming 496 fds.

  "jvm/fd_count" : 496.0,
  "jvm/fd_limit" : 16384.0,
  "rt/http/dst/id/#/io.l5d.serversets/sd/nobody/prod/packager-proxy/failure_accrual/probes" : 28,
  "rt/http/dst/id/#/io.l5d.serversets/sd/nobody/prod/packager-proxy/failure_accrual/removals" : 2,
  "rt/http/dst/id/#/io.l5d.serversets/sd/nobody/prod/packager-proxy/failure_accrual/removed_for_ms" : 2827504,
  "rt/http/dst/id/#/io.l5d.serversets/sd/nobody/prod/packager-proxy/failure_accrual/revivals" : 2,

How to reproduce:

downstream returns 5xx
retry budget exhausts
instance is marked failed

here is my linkerd config

---
admin:
  port: 9990
namers:
- kind: io.l5d.serversets
  zkAddrs:
  - host: mesos-master01of2.thebrighttag.com
    port: 2181
  - host: mesos-master02of2.thebrighttag.com
    port: 2181
  - host: mesos-master03of2.thebrighttag.com
    port: 2181
telemetry:
- kind: io.l5d.zipkin
  host: mesos-master01of2.thebrighttag.com
  port: 9410
  sampleRate: 0.001
telemetry:
- kind: io.l5d.prometheus
routers:
- protocol: http
  label: http
  dstPrefix: /host
  maxChunkKB: 16
  maxHeadersKB: 16
  maxInitialLineKB: 16
  maxRequestKB: 102400 # 100MB
  maxResponseKB: 102400 # 100MB
  compressionLevel: 9
  interpreter:
    kind: io.l5d.namerd
    dst: /#/io.l5d.serversets/sd/nobody/prod/namerd:thrift
    namespace: default
  responseClassifier:
    kind: io.l5d.retryableIdempotent5XX
  client:
    retries:
      budget:
        minRetriesPerSec: 5
        percentCanRetry: 0.5
        ttlSecs: 15
      backoff:
        kind: jittered
        minMs: 10
        maxMs: 10000
  servers:
  - port: 80
    label: prod/linkerd
    ip: 10.150.150.227
    announce:
    - /#/io.l5d.serversets/prod/linkerd
  - port: 80
    ip: 127.0.0.1
  announcers:
  - kind: io.l5d.serversets
    pathPrefix: /sd/nobody
    zkAddrs:
    - host: mesos-master01of2.thebrighttag.com
      port: 2181
    - host: mesos-master02of2.thebrighttag.com
      port: 2181
    - host: mesos-master03of2.thebrighttag.com
      port: 2181