Connection leak

What kind an issue is this?

• Bug report.

Issue description

I have a long running job on spark that makes quite a lot of reading requests to ES.

There is definitely a connection leak related to HTTP client, because

lsof -p %sparkExecutorPID% | grep TCP | grep 9200 | sort -t '>' -k2 | wc -l

shows over 1000 connections after 1 day of work. However there is only one esRDD is used at any moment of time.

Eventually this leads to too many open files error.

Log (for es-hadoop 6.2.1):

2018-10-29 03:28:13 INFO HttpMethodDirector: I/O exception (java.net.SocketException) caught when processing request: Too many open files
2018-10-29 03:28:13 INFO HttpMethodDirector: Retrying request

Version Info

OS: : Debian 8 JVM : java8 oracle Spark: 2.3.1 ES-Hadoop : tried elasticsearch-spark-20 versions 5.6.10 and 6.2.1. ES : 5.6.10

[masseyke]

@ov7a thanks for the code to reproduce this – that was really helpful. I believe I have tracked down the problem. There is a connection leak as you suspected. The connections are not getting explicitly cleaned up, but are instead cleaned up when the garbage collector reclaims the Socket objects. If you’re running with a large heap and a small limit on open files this could definitely be a problem (I never saw the number of connections get above 500, but I didn’t have a huge heap). I believe that the problem is at https://github.com/elastic/elasticsearch-hadoop/blob/master/spark/core/src/main/scala/org/elasticsearch/spark/rdd/AbstractEsRDDIterator.scala#L54 where we create a RestRepository, perform a scroll, and then never close the RestRepository. The problem seems to go away when I fix that. I’ll get a PR up sometime soon.

[masseyke]

This will be fixed in 7.16.0 and 8.0.0