OutOfMemory Errors with namerd and Consul

Moving away from https://github.com/linkerd/linkerd/issues/1361 since it describes a different scenario.

We are seeing a issue similar to the above mentioned with Namerd (v1.1.0) using the Consul Namer and the Consul Storage Backend. We run three Namerd instances behind a load balancer and we are consistently getting out-of-memory errors even after increasing the hard and soft limit in Docker and the heap size in the JVM. We’re now at 1024m of memory and nodes fail approximately every 8 hours.

We activated the admin route and were able to capture the situation in a near-to-crash slot:

• On 83% memory usage
• On 95% memory usage
• About to crash

@olix0r mentioned something was probably going wrong with the consul client.

Here’s a namerd.yml configuration that could help to reproduce the issue:

admin:
  ip: 0.0.0.0
  port: 9991
interfaces:
  - kind: io.l5d.thriftNameInterpreter
    ip: 0.0.0.0
    port: 4100
  - kind: io.l5d.httpController
    ip: 0.0.0.0
    port: 4180
namers:
  - kind: io.l5d.consul
    host: X.X.X.X
    prefix: /consul
    includeTag: true
storage:
  kind: io.l5d.consul
  experimental: true
  host: X.X.X.X
  pathPrefix: /namerd/dtabs

[hawkw]

Well, I have a suspect:

[hawkw]

I’m closing this due to inactivity. @eduponte, if you consider seeing this issue on the latest Namerd release, please re-open.