Memory leak when client cache is evicting

First reported by @Ashald and @edio.

In a demo setup where the number of destinations exceeds the bindingCache, the linkerd process leaks memory as it adds/removes clients to/from the cache.

The easiest way to reproduce is with this docker-compose setup:

client-memory-issue.zip

Start the demo with:

docker-compose build && docker-compose up -d

Then load the linkerd admin dashboard on docker port 9990. The initial setup has 10 destinations and a client cache size of 10. You can see from the dashboard that the heap size for the process hovers at ~85mb.

Next stop the demo:

docker-compose stop && docker-compose rm -vf

And edit linkerd.yml to shrink the binding cache size from 10 to 9. Then restart the demo and reload the admin dashboard. You’ll see the heap size start to creep up over time. Left running for about 30 minutes, heap will be ~400mb.

[chrisgoffinet]

I have confirmed this is a real leak. This exists in the latest build as well. I’ve taken a heap dump and working on a patch to fix this.

[koiuo]

Frankly speaking, I have hard time recalling any details, but as @ashald mentioned, we have not been observing any related symptoms for quite a while already. I guess this issue can be closed.