High memory usage on Kubernetes (~400MB with close to zero traffic)

Might be related to #1343.

I’m running two linkerd instances: one as a sidecar instance, and the other one as an edge proxy.

            +-----------------------------------+                  +---------------------------------+
            |                                   |                  |                                 |
            |                                   |                  |                                 |
            |                                   |                  |  +-----------+       +--------+ |
            |          LinkerD Edge             |                  |  |           |       |        | |
+----------->                                   +---------------------> LinkerD   +------->  App   | |
            |                                   |                  |  +-----------+       +--------+ |
            |                                   |                  |                                 |
            |                                   |                  |                                 |
            +-----------------------------------+                  +---------------------------------+

Those two instance seem to take way too much memory, even there is little to no traffic. I made a test where I hit the linkerd edge instance (itself hitting the linkerd sidecar instance) at a rate of 2 requests per second, and I’m getting a memory usage of 300-400 MB (which obviously isn’t viable):

LinkerD sidecar instance memory usage

LinkerD edge instance memory usage

I’m not sure this is normal, but according to some other benchmarks I’ve seen, I think the memory footprint of LinkerD is supposed to be much lower. Any idea of what could be the cause?

Additional details:

• Environment: OpenShift v1.5.1 running Kubernetes v1.5.2
• LinkerD version: 1.3.0
• Configuration files: see here
• JVM stats from /admin/metrics.json: see here

The Kubernetes cluster I’m running LinkerD on will be upgraded to a more recent Kubernetes version shortly, I’ll let you know if that changes anything.

Thanks!

[siggy]

Hi @christophetd, thanks for the detailed report. We’re are currently investigating possibly related memory usage/leak issues, follow along at #1685 and #1690. Stay tuned.

[wmorgan]

@christophetd I’m going to close this issue, since I believe we’ve fixed the underlying problem. If you still see this on 1.3.7 or beyond, please feel free to reopen.