Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Off-heap memory leak in Spark < 2.1: PoolThreadCaches aren't getting freed?

See original GitHub issue

Expected behavior

PoolThreadCache should be freed when thread it is bound to terminates.

Actual behavior

We observed off-heap memory leak in our application that is run in Spark 1.6.2 on Amazon EMR 4.8.0. Analyzing driver’s heap dump I noticed that PoolThreadLocalCache’s caches counter doesn’t decrease which makes me thinking that PoolThreadCache is not getting freed: At the same time there are not that many threads:

Steps to reproduce

I don’t think it is specific to our application but rather to Spark/Netty itself.

Minimal yet complete reproducer code (or URL to code)

I can provide heap dumps if needed.

Netty version

4.0.29.Final

JVM version (e.g. `java -version`)

java version “1.8.0_45” Java™ SE Runtime Environment (build 1.8.0_45-b14) Java HotSpot™ 64-Bit Server VM (build 25.45-b02, mixed mode)

OS version (e.g. `uname -a`)

Amazon (emr-4.8.0), Linux 4.4.11-23.53.amzn1.x86_64 #1 SMP Wed Jun 1 22:22:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Issue Analytics

State:
Created 6 years ago
Comments:9 (6 by maintainers)

Top GitHub Comments

1reaction

normanmaurercommented, Jun 6, 2017

Then maybe you should create an issue in spark ?

On 6. June 2017 at 20:28:25, Yauheni Audzeichyk (notifications@github.com) wrote:

@Scottmitch https://github.com/scottmitch unfortunately, I can not upgrade to a newer version of Netty as Spark on EMR comes bundled with it. Also that property is not available in 4.0.29.Final.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/netty/netty/issues/6819#issuecomment-306575848, or mute the thread https://github.com/notifications/unsubscribe-auth/AAa0QgUoNu8bbvpg1aiPGpRLwev-mwvIks5sBZpJgaJpZM4NxPV3 .

0reactions

Scottmitchcommented, Jun 8, 2017

@yaud - great thanks for closing the loop!