Cluster keeps eating up memory and not freeing it

CrateDB version: 1.0.2

JVM version:

openjdk version "1.8.0_121"
OpenJDK Runtime Environment (build 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13)
OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)

OS version / environment description: Ubuntu 16.04.1 LTS

Problem description: We are seeing that the crate cluster just keeps allocating memory, especially under heavy querying load. The cluster has almost no ingress load. We are monitoring all cluster nodes with prometheus and as you can see in the picture below on every cpu usage spike, total memory free drops. We have to restart certain nodes (e.g. on 2/21 at ~11 o’clock) before they run into an OOM Exception and the cluster becomes eventually unhealthy. All nodes, except 2, in the cluster have 60gb ram, max heap is set to 30gb for crate. The two smaller nodes have 14gb ram and 7gb heap for crate. These two nodes have been running for some months and we had no problems with them so memory freeing seems to be fine on them!

The cluster holds only 6Mio rows at 5.8gb.

[seut]

@lukaskroepfl jfyi: I’m currently investigating and fixing memory leaks which are happening on error handling. maybe this is related to your issue as well. will notify you when fixes are merged/available…

[jodok]

closing due to inactivity. feel free to reopen if the problem reappears.