Native memory excessive consumption and native memory leaks

Description

When running Teku with -Xmx1G the java process resident memory tend to be of ~2.5G which looks like an overhead

When running Teku with e.g. -Xmx1G, the total JVM memory consumption is estimated to be around 1.5G (including ByteBuffers which are mostly Netty owned). It’s also expected for RocksDB to consume around 200Mb with the current defaults:

• 128Mb for write buffer
• 8Mb for block cache
• Other native structures

Thus we having ~0.8G of unknown native memory consumption. RocksDB is the major native lib used in Teku so it is the primary suspect.

Here is the process memory consumption over time:

From the graph it doesn’t looks like the memory is leaking

Acceptance Criteria

Teku process with -Xmx1G shouldn’t consume more than 2G

Steps to Reproduce (Bug)

Run Teku node with -Xmx1G on Onyx (or other) testnet for about 2-3 hours

Expected behavior: process consumes < 2Gb

Actual behavior: process consumes 2.5G

Frequency: Always

Versions (Add all that apply)

commit d98d802cb416819a22081be693ffb130c052e425 (HEAD -> master, origin/master, origin/HEAD)
Author: Adrian Sutton <[e-mail]>
Date:   Wed Jul 1 15:03:46 2020 +1000

[benjaminion]

@tzapu This is most like related to https://github.com/ConsenSys/teku/issues/3495 - (excessive memory consumption by RocksDB). We are actively looking at alternative DBs at the moment, and will hopefully have news soon. It would definitely be good to resolve this!

[ajsutton]

Ah great to hear you’ve found the issue. j9 isn’t particularly common and IBM tends to focus on big iron server deployments of it so I’m not entirely surprised it winds up using a lot of memory. Likely it could be controlled by setting the right options but it may not be setting defaults as intelligently as Hotspot.