spring boot redeploy in tomcat causing a permgen memory leak

I am currently trying to resolve an issue we’re seeing with redeploying a spring boot application that looks to be a problem in the spring boot end not ours?

Links:

• Stack overflow question: https://stackoverflow.com/questions/70174850/spring-boot-tomcat-redeploy-permgen-leak-due-to-logging
• Git repo: https://github.com/webbres/spring-boot-permgen-issue/tree/master/debugging

Config: Java 11 OS: Windows & Centos Spring boot: 2.5 Tomcat: 9.0.27

---

The problem/bug:

When deploying on tomcat 9 using java 11 reloading an application results in a PermGen memory leak with the classloader for the application not being GCd. After adding additional configuration only weak and soft references are found in the heap.

Summary of findings:

Reloading a spring boot application deployed as a war in tomcat 9 results in a PermGen memory leak (reported by the find leaks check and checked in VisualVM). In our actual application there is a strong reference to a log4j JMX bean which is removed when running tomcat with -Dlog4j2.disable.jmx=true though even then the org.apache.catalina.loader.ParallelWebappClassLoader for the application is marked as DESTROYED but is not garbage collected.

I attempted to recreate the issue on a simple spring boot starter (see repo linked). Here I found strong references to logging shutdown hooks which are removed when adding logging.register-shutdown-hook=false to the application.properties. Though after adding this property like above the classloader still isn’t being garbage collected.

Cheers

Sam

[wilkinsona]

As far as I can tell, this problem is specific to Java 11.

Running with Java 8, when an application is undeployed and GC is forced, all of its classes are unloaded and its ClassLoader is garbage collection. Running with Java 11, the ClassLoader is never garbage collected. In both cases, there are no strong references to the ClassLoader. In Java 8, classes loaded by the class loader are only referenced weakly. In Java 11, some of the classes loaded by the class loader are referenced softly. The presence of soft references is key to the difference in behaviour.

Generally speaking, weak references will be cleaned up whenever garbage collection is forced. By contrast, soft references will only be cleaned up when the JVM is under memory pressure. Examining in YourKit the heap of a basic Spring Boot web application running on Java 11 reveals soft references from com.sun.beans.introspect.ClassInfo to various Spring and application classes. These ClassInfo instances are being created by java.beans.Introspector via org.springframework.beans.CachedIntrospectionResults.

CachedIntrospectionResults used to call java.beans.Introspector.flushFromCaches(Class<?>) immediately after introspecting a class. This was removed as, due to changes in the JDK, it was no longer needed. It looks to me like it may be needed again following these changes in the JDK which introduced ClassInfo and a soft reference cache.

[wilkinsona]

The fix has already been made in the JDK but only in Java 16 and later. For that fix to be effective Framework’s call to flushFromCaches needs to be reinstated as far as I can tell. For Java 11, I’m not sure what can be done. At best, a change here would need to be a reflective hack as just calling flushFromCaches won’t help. The ideal would be a backport of JDK-8231454 to Java 11.