Thingsboard crashes every couple of days. java.lang.OutOfMemoryException Java Heap Space

Hello. I have been working with thingsboard for a while. I am publishing data to thingsboard through a custom application that acts as a gateway (collects data from three devices currently) and pushes data to thingsboard after about every 8 seconds via mqtt. It works fine for a couple of days but then thingsboard crashes. The thingsboard service shows as “Active (running)” but when i try to open the thingsboard UI through the browser it does not load. My application that pushes data to thingsboard also stops publishing data as it is unable to establish a connection through mqtt. I have to restart the thingsboard service everytime this happens. Once the service is restarted, the data publishing resumes from my application. I have tried using thingsboard with the embedded HSQLDB, Cassandra and PostgreSQL but the same OutOfMemoryException : Java Heap Space seems to cause a crash. The only difference is that with Cassandra the crash happens after about three days whereas with PostgreSQL and the embedded HSQLDB, it happens within a day or two.

My thingsboard instance is hosted on a digital ocean droplet with 4GB RAM/ 2 CPUs running Ubuntu . So far i have tried increasing the memory to 1GB in the /etc/thingsboard/thingsboard.conf file by adding this line: export JAVA_OPTS="$JAVA_OPTS -Dplatform=deb -Xms1024M -Xmx1024M", but the issue persists. I restarted the thingsboard service and monitored the memory usage of thingsboard using top and found that within 24 hours the usage increased from 17.8% to 38.0%.

Here are the ERRORs from the thingsboard.log file that occurred during two different crashes. The errors are always Java heap space errors:

2018-01-08 02:12:21,932 [nioEventLoopGroup-4-1] ERROR i.n.u.c.D.rejectedExecution - Failed to submit a listener notification task. Event loop shut down?
2018-01-08 02:24:18,326 [pool-23-thread-1] ERROR o.t.s.a.p.PluginProcessingContext - **Critical error: Java heap space**

2018-01-09 01:08:00,507 [nioEventLoopGroup-5-9] ERROR o.t.s.t.mqtt.MqttTransportHandler - [mqtt11829] Unexpected Exception
2018-01-09 01:08:02,479 [http-nio-0.0.0.0-8080-exec-1] ERROR o.a.c.c.C.[.[.[.[dispatcherServlet] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Handler processing failed; nested exception is **java.lang.OutOfMemoryError: Java heap space**] with root cause
  

[cchaz003]

To add to that, how would you clean up the data file?

[koo9]

@cchaz003 I am with you there. we just started evaluation on TB. first looks, it meet all our need but this could be a show stopper. I thought the problem is that the instance does not have enough memory to run TB but 8G should be sufficient. out of memory problem is common for java app. not a fan.

would love to hear any success story with TB.