Linkerd initial memory usage higher in latest versions

Issue Type:

• Bug report

What happened:

We are trying to upgrade from linkerd 1.4.1 to 1.5.2. In doing so, we noticed the Java RSS memory usage was larger on the new instances, even though we have not changed any memory settings or linkerd yaml.

What you expected to happen:

We expected memory usage to be consistent between linkerd versions, with possible minor differences.

How to reproduce it (as minimally and precisely as possible):

Environment:

JVM_HEAP_MAX=1024M
JVM_HEAP_MIN=1024M
FINAGLE_WORKERS=24
LOCAL_JVM_OPTIONS=-XX:NativeMemoryTracking=summary

Startup linkerd on a box with JDK 8 (I’m using openjdk), and note the process-id. No need to run any traffic through it. Run command jcmd <your-pid> VM.native_memory summary and note the total memory committed to linkerd. In 1.5.2, this is what I see:

Native Memory Tracking:

Total: reserved=3166475KB, committed=2079147KB
-                 Java Heap (reserved=1048576KB, committed=1048576KB)
                            (mmap: reserved=1048576KB, committed=1048576KB)

-                     Class (reserved=1101695KB, committed=61231KB)
                            (classes #9424)
                            (malloc=1919KB #13739)
                            (mmap: reserved=1099776KB, committed=59312KB)

-                    Thread (reserved=83571KB, committed=83571KB)
                            (thread #82)
                            (stack: reserved=83204KB, committed=83204KB)
                            (malloc=272KB #418)
                            (arena=95KB #160)

-                      Code (reserved=50449KB, committed=3585KB)
                            (malloc=529KB #2000)
                            (mmap: reserved=49920KB, committed=3056KB)

-                        GC (reserved=21756KB, committed=21756KB)
                            (malloc=18336KB #308)
                            (mmap: reserved=3420KB, committed=3420KB)

-                  Compiler (reserved=249KB, committed=249KB)
                            (malloc=119KB #379)
                            (arena=130KB #4)

-                  Internal (reserved=800308KB, committed=800308KB)
                            (malloc=800308KB #63799)

-                    Symbol (reserved=13036KB, committed=13036KB)
                            (malloc=11621KB #90207)
                            (arena=1415KB #1)

-    Native Memory Tracking (reserved=2688KB, committed=2688KB)
                            (malloc=13KB #147)
                            (tracking overhead=2675KB)

-               Arena Chunk (reserved=253KB, committed=253KB)
                            (malloc=253KB)

-                   Unknown (reserved=43896KB, committed=43896KB)
                            (mmap: reserved=43896KB, committed=43896KB)

Of note, “Internal” has reserved 800MB on top of the 1000MB heap. I don’t have an equivalent output from linkerd 1.4.1, but I know the RSS total was very close to the heap size (not double the heap size).

Environment

I’ll work on a minimal reproducible config example. For now, I’ll say that we’re running linkerd, connecting to namerd, which resolves mesos marathon apps.

[dadjeibaah]

Managed to track this down and found out that this memory increase is caused by this netty issue and has been fixed since Netty 4.1.31. The finagle repo also has an issue that reported a similar problem here and they also have recently picked up the Netty fix.

It looks like the most recent version of Finagle (18.12.0) corrects this increased non-heap memory and upgrading Linkerd to use Finagle 18.12.0 will ultimately fix this issue. Yay to library dependencies 😅!

Issue references: https://github.com/twitter/finagle/issues/738 https://github.com/netty/netty/issues/8159

[DukeyToo]

Full reproducible example is here: https://github.com/DukeyToo/linkerd-docker-example - thanks again @ganasubrgit for putting that together.

To reproduce, change linkerd version as needed in the docker compose file, then run:

docker-compose up -d
docker exec -it docker-linkerd_l5d_1 /bin/bash
apt-get update && apt-get install -y openjdk-8-jdk
jcmd 1 VM.native_memory summary