Memory usage

I am noticing a big increase in memory usage between the releases 3.1.3 and 4.0.0 and above.

Our app is running on node 0.12.6 and we create our elasticsearch client with the following properties:

var esClient = new elasticsearch.Client({
  hosts: localhost:9200,
  apiVersion: "1.3",
  minSockets: 10,
  maxSockets: 10
});

When running on 3.1.3 we converge around 350-400MB. When upgrading to 4.0.0 or any later release the memory just seems to grow until hitting the v8-limit (1,4GB).

[andrewstoker]

@spalger Is this leaky behaviour still being investigated in the newer versions beyond 3.1.4? We are on node v4.4.5 and with any elasticsearch.js client above 4.x, the memory usage is far higher, leaking and behaving much differently as observed by @dylancwood. See below two charts.

V3.1.4 client:

V11.0.1 client:

Both are with the following client configuration (hosts commented out) :

elasticsearch: {
        hosts:           [
            'http://xxxxx:9200',
            'http://xxxxx:9200',
            'http://xxxxx:9200'
        ],
        minSockets:             10,
        maxSockets:             100
    }

[dylancwood]

I think that I am experiencing the same issue. We are running a relatively thin proxy layer between our mobile clients and elastic search. Each node of that layer is receiving around 300 requests per minute. When we upgraded to Node 4+, we ran into max-stack depth exceeded errors with elasticsearch-js 3.1.3, so we upgraded to the latest version.

Memory consumption went up about four-fold with just this little change. To confirm that elasticsearch-js is responsible, I forked the repo and fixed the max-stack depth exceeded issue by pulling in a change from later versions of elasticsearch-js (https://github.com/dylancwood/elasticsearch-js/blob/3-1-3-speed/src/lib/log.js#L66).

Below are graphs of memory consumption, throughput and response time for two servers. Both servers are running Node 4.3.0, and were inundated with as many requests as they could handle for about 15 minutes (hence the poor response time). The only difference is that the red one is using elasticsearch-js 4.0.0 and the yellow one is using my patched fork of elasticsearch-js 3.1.3. Note that the response time is actually better on the 4.0.0 version 👍, but 700MB of memory is a high price to pay. Also, I cannot explain why the response time is faster on the 4.0.0 server, but the throughput is lower…

@spalger: I hope you’ll consider this to be solid evidence. FTR, we use only three methods of this package: mget, search and indices.analyze.