ClassNotFoundException can be thrown for classes in nested jars when under GC pressure

Overview

A very simple Spring Boot based application locks up on the very first HTTP request with java.lang.NoClassDefFoundError present in the logs. Killing the application also causes java.lang.NoClassDefFoundError in the logs.

The matrix below outlines when the locking happens by the version and the execution environment.

Spring Boot	Linux	Windows	OpenShift
2.5.14	Works	Works	Works
2.6.8	Works	Works	Works
2.6.9+	Works	Works	Locks
2.7.0	Works	Works	Works
2.7.1+	Works	Works	Locks

Normal execution

Here is how a normal execution looks.

> oc run image-test --rm -it --image=registry/angular-starter:1.0.0-SNAPSHOT --image-pull-policy="Always" --command -- /bin/sh
If you don't see a command prompt, try pressing enter.
sh-4.4$ java -jar /app/lib/starter-angular-app-1.0.0-SNAPSHOT-exec.jar &
[1] 7
  .   ____          _            __ _ _
/\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot ::                (v2.6.8)
 
2022-07-24 17:11:13.850  INFO 7 --- [           main] com.baml.starter.angular.web.WebApp      : Starting WebApp using Java 11.0.15 on image-test with PID 7 (/app/lib/starter-angular-app-1.0.0-SNAPSHOT-exec.jar started by ? in /home/app)
2022-07-24 17:11:13.853  INFO 7 --- [           main] com.baml.starter.angular.web.WebApp      : No active profile set, falling back to 1 default profile: "default"
2022-07-24 17:11:19.854  INFO 7 --- [           main] o.s.b.web.embedded.netty.NettyWebServer  : Netty started on port 8080
2022-07-24 17:11:19.863  INFO 7 --- [           main] com.baml.starter.angular.web.WebApp      : Started WebApp in 7.61 seconds (JVM running for 9.215)
 
sh-4.4$ curl -v http://localhost:8080/api/status
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
> GET /api/status HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.61.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: application/hal+json
< Content-Length: 86
< 
* Connection #0 to host localhost left intact
{"_links":{"self":{"href":"/api/status"}},"apiVersion":"1.0.0-SNAPSHOT","status":"UP"}
sh-4.4$ kill %1
[1]+  Done(143)               java -jar /app/lib/starter-angular-app-1.0.0-SNAPSHOT-exec.jar

Stuck execution

Here is how the stuck request looks. I had to press ^C to abort the curl run.

> oc run image-test --rm -it --image=registry/angular-starter:1.0.0-SNAPSHOT --image-pull-policy="Always" --command -- /bin/sh
If you don't see a command prompt, try pressing enter.
sh-4.4$ java -jar /app/lib/starter-angular-app-1.0.0-SNAPSHOT-exec.jar &
[1] 7
  .   ____          _            __ _ _
/\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot ::                (v2.6.9)
 
2022-07-24 16:53:02.884  INFO 7 --- [           main] com.baml.starter.angular.web.WebApp      : Starting WebApp using Java 11.0.15 on image-test with PID 7 (/app/lib/starter-angular-app-1.0.0-SNAPSHOT-exec.jar started by ? in /home/app)
2022-07-24 16:53:02.887  INFO 7 --- [           main] com.baml.starter.angular.web.WebApp      : No active profile set, falling back to 1 default profile: "default"
2022-07-24 16:53:09.586  INFO 7 --- [           main] o.s.b.web.embedded.netty.NettyWebServer  : Netty started on port 8080
2022-07-24 16:53:09.679  INFO 7 --- [           main] com.baml.starter.angular.web.WebApp      : Started WebApp in 8.394 seconds (JVM running for 9.951)
 
sh-4.4$ curl -v http://localhost:8080/api/status
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
> GET /api/status HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.61.1
> Accept: */*
> 
Exception in thread "reactor-http-epoll-2" java.lang.NoClassDefFoundError: io/netty/buffer/PoolArena$1
        at io.netty.buffer.PoolArena.freeChunk(PoolArena.java:248)
        at io.netty.buffer.PoolThreadCache$MemoryRegionCache.freeEntry(PoolThreadCache.java:432)
        at io.netty.buffer.PoolThreadCache$MemoryRegionCache.free(PoolThreadCache.java:396)
        at io.netty.buffer.PoolThreadCache$MemoryRegionCache.free(PoolThreadCache.java:388)
        at io.netty.buffer.PoolThreadCache.free(PoolThreadCache.java:254)
        at io.netty.buffer.PoolThreadCache.free(PoolThreadCache.java:245)
        at io.netty.buffer.PoolThreadCache.free(PoolThreadCache.java:218)
        at io.netty.buffer.PooledByteBufAllocator$PoolThreadLocalCache.onRemoval(PooledByteBufAllocator.java:542)
        at io.netty.buffer.PooledByteBufAllocator$PoolThreadLocalCache.onRemoval(PooledByteBufAllocator.java:503)
        at io.netty.util.concurrent.FastThreadLocal.remove(FastThreadLocal.java:257)
        at io.netty.util.concurrent.FastThreadLocal.removeAll(FastThreadLocal.java:67)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:1050)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.ClassNotFoundException: io.netty.buffer.PoolArena$1
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
        at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:151)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        ... 15 more
^C
sh-4.4$ kill %1
Exception in thread "SpringApplicationShutdownHook" java.lang.NoClassDefFoundError: ch/qos/logback/classic/spi/ThrowableProxy
        at ch.qos.logback.classic.spi.LoggingEvent.<init>(LoggingEvent.java:119)
        at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:419)
        at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
        at ch.qos.logback.classic.Logger.log(Logger.java:765)
        at org.apache.commons.logging.LogAdapter$Slf4jLocationAwareLog.warn(LogAdapter.java:447)
        at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1070)
        at org.springframework.boot.web.reactive.context.ReactiveWebServerApplicationContext.doClose(ReactiveWebServerApplicationContext.java:147)
        at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:1021)
        at org.springframework.boot.SpringApplicationShutdownHook.closeAndWait(SpringApplicationShutdownHook.java:145)
        at java.base/java.lang.Iterable.forEach(Iterable.java:75)
        at org.springframework.boot.SpringApplicationShutdownHook.run(SpringApplicationShutdownHook.java:114)
        at java.base/java.lang.Thread.run(Thread.java:829)
 
[1]+  Done(143)               java -jar /app/lib/starter-angular-app-1.0.0-SNAPSHOT-exec.jar

This particular application uses WebFlux to serve the request. I tried to convert it to WebMVC. It also gets stuck with java.lang.NoClassDefFoundError albeit in a different class.

I did a quick diff between 2.6.8 and 2.6.9. I see some changes in org.springframework.boot.loader.jar package that may be relevant here. I’m referring to this commit. For what it’s worth.

[ayashkov]

I’m doing essentially this: https://github.com/ayashkov/spring-boot-31853. Several things though:

1. I could not make this specific code fail. Still trying to modify it to trigger the failure.
2. The whole thing seems to be extremely sensitive to the composition of the final JAR. E.g. when I try to make certain changes to the code that fails, e.g. consolidating or renaming modules, removing HAL processing, it stops failing.

I will keep working on it for a few days more, hopefully with better outcome.

[ayashkov]

The issue is reproducible in Docker. Please see https://github.com/ayashkov/spring-boot-31853.