Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Linkerd initial memory usage higher in latest versions

See original GitHub issue

Issue Type:

  • Bug report

What happened:

We are trying to upgrade from linkerd 1.4.1 to 1.5.2. In doing so, we noticed the Java RSS memory usage was larger on the new instances, even though we have not changed any memory settings or linkerd yaml.

What you expected to happen:

We expected memory usage to be consistent between linkerd versions, with possible minor differences.

How to reproduce it (as minimally and precisely as possible):



Startup linkerd on a box with JDK 8 (I’m using openjdk), and note the process-id. No need to run any traffic through it. Run command jcmd <your-pid> VM.native_memory summary and note the total memory committed to linkerd. In 1.5.2, this is what I see:

Native Memory Tracking:

Total: reserved=3166475KB, committed=2079147KB
-                 Java Heap (reserved=1048576KB, committed=1048576KB)
                            (mmap: reserved=1048576KB, committed=1048576KB)

-                     Class (reserved=1101695KB, committed=61231KB)
                            (classes #9424)
                            (malloc=1919KB #13739)
                            (mmap: reserved=1099776KB, committed=59312KB)

-                    Thread (reserved=83571KB, committed=83571KB)
                            (thread #82)
                            (stack: reserved=83204KB, committed=83204KB)
                            (malloc=272KB #418)
                            (arena=95KB #160)

-                      Code (reserved=50449KB, committed=3585KB)
                            (malloc=529KB #2000)
                            (mmap: reserved=49920KB, committed=3056KB)

-                        GC (reserved=21756KB, committed=21756KB)
                            (malloc=18336KB #308)
                            (mmap: reserved=3420KB, committed=3420KB)

-                  Compiler (reserved=249KB, committed=249KB)
                            (malloc=119KB #379)
                            (arena=130KB #4)

-                  Internal (reserved=800308KB, committed=800308KB)
                            (malloc=800308KB #63799)

-                    Symbol (reserved=13036KB, committed=13036KB)
                            (malloc=11621KB #90207)
                            (arena=1415KB #1)

-    Native Memory Tracking (reserved=2688KB, committed=2688KB)
                            (malloc=13KB #147)
                            (tracking overhead=2675KB)

-               Arena Chunk (reserved=253KB, committed=253KB)

-                   Unknown (reserved=43896KB, committed=43896KB)
                            (mmap: reserved=43896KB, committed=43896KB)

Of note, “Internal” has reserved 800MB on top of the 1000MB heap. I don’t have an equivalent output from linkerd 1.4.1, but I know the RSS total was very close to the heap size (not double the heap size).


I’ll work on a minimal reproducible config example. For now, I’ll say that we’re running linkerd, connecting to namerd, which resolves mesos marathon apps.

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Comments:7 (3 by maintainers)

github_iconTop GitHub Comments

dadjeibaahcommented, Dec 14, 2018

Managed to track this down and found out that this memory increase is caused by this netty issue and has been fixed since Netty 4.1.31. The finagle repo also has an issue that reported a similar problem here and they also have recently picked up the Netty fix.

It looks like the most recent version of Finagle (18.12.0) corrects this increased non-heap memory and upgrading Linkerd to use Finagle 18.12.0 will ultimately fix this issue. Yay to library dependencies 😅!

Issue references:

DukeyToocommented, Nov 23, 2018

Full reproducible example is here: - thanks again @ganasubrgit for putting that together.

To reproduce, change linkerd version as needed in the docker compose file, then run:

docker-compose up -d
docker exec -it docker-linkerd_l5d_1 /bin/bash
apt-get update && apt-get install -y openjdk-8-jdk
jcmd 1 VM.native_memory summary
Read more comments on GitHub >

github_iconTop Results From Across the Web

Benchmarking Linkerd and Istio: 2021 Redux
Linkerd's data plane is heavier than in 2.10, reporting max CPU usage of 36ms (previously 11ms) and max memory usage to 26mb (previously...
Read more >
Linkerd: high memory usage on Kubernetes
Linkerd starting memory use is typically around 100-120mb and can go up with high volume of traffic but our experience is that it...
Read more >
Benchmarking Linkerd and Istio
Starting with the control plane, we see that Istio's control plane usage averaged 837mb, about 2.5x Linkerd's control plane memory consumption ...
Read more >
Proxy Configuration | Linkerd
Linkerd provides a set of annotations that can be used to override the data plane proxy's ... Maximum amount of Memory that the...
Read more >
High CPU and memory usage with frequent k8s state changes
Testing linkerd in my perf environment I noticed that namerd cpu and memory spike when my pods frequently cycle their readiness state. When...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found