MySQL `ColumnDefinition` might pile up in young generation due to GC nepotism

When running with GC configurations that result in early promotion, some of the vert.x sqlclient and/or mysqlclient request specific classes might not be freed up during young generation garbage collection even though they’re not in use.

To replicate, run the Baeldung article code with Quarkus Native, or Quarkus JVM mode configured with Parallel GC. Under these circumstances, objects keep building up the young generation and only get garbage collected upon a full GC. This can create memory graphs like this one with Quarkus Native:

Getting heap dumps when close to full GC can result in Eclipse MAT displaying the following class histogram of unreachable classes. There’s a high number of io.vertx.mysqlclient.impl.protocol.ColumnDefinition instances:

To get a better picture of the object graph surrounding io.vertx.mysqlclient.impl.protocol.ColumnDefinition, you can build Quarkus Native with epsilon GC and get a heap dump shortly after introducing the load. Using VisualVM (which doesn’t discard unreachable objects), you can see the object graph:

A case of GC nepotism might be happening here. Some of the objects might be live longer than others and might get promoted to the tenure while keeping references to other short lived objects. Even though the objects that have been promoted to tenure might be dead, they won’t be garbage collected until there’s a full GC and since they keep references to short lived objects in the young generation, those can’t be collected either.

Nulling references to break the graph when short lived objects are no longer in use should help.

Version

Vert.x 4.3.2

Do you have a reproducer?

Run benchmark in the Baeldung blog post either with Quarkus Native or Quarkus JVM mode with Parallel GC.

Steps to reproduce

1. git clone https://github.com/eugenp/tutorials
2. cd tutorials/quarkus-modules/quarkus-vs-springboot/quarkus-project
3. ./mvnw clean package -Dnative
4. Add -XX:+UseParallelGC -verbose:gc to start_jvm.sh
5. docker run --name mysqldb --network=host -p 3306:3306 -e MYSQL_ROOT_PASSWORD=root -e MYSQL_DATABASE=baeldung -d mysql:5.7.38 --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
6. ./start_jvm.sh
7. In another folder/machine git clone https://github.com/eugenp/tutorials
8. cd tutorials/quarkus-modules/quarkus-vs-springboot/wrk
9. ./run_test_wrk.sh (you need wrk in path or modify script)

Extra

• Java 17 or GraalVM 22.2.0 (for native).

[tqvarnst]

Just to be clear the sawtooth-shaped graphs are very common for most well-behaved java applications, but the more we can avoid full GC the better. Just for anyone else seeing this having a shape like this is not necessarily a bad thing and to understand if it’s a problem it’s much more important to check the GC logs. Ultimately if near real-time behavior is what you want and want to avoid full GCs altogether, I would recommend using a parallel GC (or pauseless) and using multiple cores for your application (single core applications do not benefit from parallel GC).

[tsegismont]

@galderz @franz1981 I failed to replicate the used heap shape you observed.

I have changed the startup script (added -XX:+UseParallelGC and tried different max heap sizes). I have tried with OpenJDK 17 and 11.

The JVM output indicates only minor collections (e.g. Pause Young (Allocation Failure) 683M->16M(2032M) 0,387ms)

Have you added any other parameter to tune GC?