7.0.0-beta6 memory issues (regression)

A test task runs over 310,000 independent test executions. In 6.14.3 this completes in about 2.5 minutes. In 7.0.0-beta6 this slows down to a crawl at 200k tests, appearing to GC thrash as small batches of tests complete.

In the past, I’ve seen this occur due to reporting keeping parameters in-memory. The project uses a workaround to replace test parameters with their toString version, as these are too heavy for a large run. I suspect there is some similar per-test memory grab by the reporter.

The next step is probably to take a heap dump and determine what is held. Before running too far into this, do you know what changes were made that might hint at what to look for?

./gradlew :caffeine:strongKeysAndStrongValuesAsyncCaffeineTest

[ben-manes]

In today’s round of optimizations, I was able to drop a test suite task from 315MB (8.6M objects) down to 22MB (0.5M objects). The noticeable pauses from a full GC also disappeared.

There are roughly 30 test suites of equal size (140k tests per suite) in order to reduce the report size, as a single test task failed due to the memory overhead. Previously I had already optimized by aggressively removing test parameters and expected exceptions if successful and shortening + dedup’ing test names in the report. The GC is also tuned.

4.2M tests executing in parallel on TravisCI with 512mb heap now takes ~1 hour.

[ben-manes]

@krmahadevan It did seem a little better by performing more tests in the stresser. Since there are still places that retain all of them, e.g. ResultMap, unfortunately you have to fix all before we feel the impact.

If you feel that you have to wait on me and get blocked, you can perform my steps:

1. Build the local jar (gradlew jar)
2. Download testng-stresser and run the project.
   • Update the build to point to your local jar
   • I added maxHeapSize = '256m' locally to fail faster.
3. Run gradlew cleanTest test until it slows down
4. Create a heap dump, e.g. jmap -dump:live,format=b,file=/Users/ben/Downloads/1.hprof "$(pgrep -f GradleWorkerMain | head -n 1)"
5. Open in your favorite tool (JVisualVM, YourKit, JProfiler). The latter two are free for OSS and are great.