Performance degradation when using JDK 11 and log4j2

Hi folks,

Thanks for this great lib. I found an issue and would like to share it with you.

With JDK 11 and multi-release open, the log4j2 lib will use the implementations with the Java9+ specific classes. This can cause a great performance degradation (roughly 30% in our service) because Armeria tries to create a new Logger on every request which will cause a slow deep stack walk initiated by log4j2.

The typical stack trace is like this:

"armeria-common-worker-nio-2-3" #35 daemon prio=5 os_prio=31 cpu=5581.88ms elapsed=20.00s tid=0x00007fee5d0ea000 nid=0x7d03 runnable  [0x0000700010ab0000]
   java.lang.Thread.State: RUNNABLE
        at java.lang.StackStreamFactory$AbstractStackWalker.fetchStackFrames(java.base@11.0.2/Native Method)
        at java.lang.StackStreamFactory$AbstractStackWalker.fetchStackFrames(java.base@11.0.2/StackStreamFactory.java:386)
        at java.lang.StackStreamFactory$AbstractStackWalker.getNextBatch(java.base@11.0.2/StackStreamFactory.java:322)
        at java.lang.StackStreamFactory$AbstractStackWalker.peekFrame(java.base@11.0.2/StackStreamFactory.java:263)
        at java.lang.StackStreamFactory$AbstractStackWalker.hasNext(java.base@11.0.2/StackStreamFactory.java:351)
        at java.lang.StackStreamFactory$StackFrameTraverser.tryAdvance(java.base@11.0.2/StackStreamFactory.java:593)
        at java.util.stream.ReferencePipeline.forEachWithCancel(java.base@11.0.2/ReferencePipeline.java:127)
        at java.util.stream.AbstractPipeline.copyIntoWithCancel(java.base@11.0.2/AbstractPipeline.java:502)
        at java.util.stream.AbstractPipeline.copyInto(java.base@11.0.2/AbstractPipeline.java:488)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(java.base@11.0.2/AbstractPipeline.java:474)
        at java.util.stream.FindOps$FindOp.evaluateSequential(java.base@11.0.2/FindOps.java:150)
        at java.util.stream.AbstractPipeline.evaluate(java.base@11.0.2/AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.findFirst(java.base@11.0.2/ReferencePipeline.java:543)
        at org.apache.logging.log4j.util.StackLocator.lambda$getCallerClass$6(StackLocator.java:59)
        at org.apache.logging.log4j.util.StackLocator$$Lambda$10/0x0000000800063040.apply(Unknown Source)
        at java.lang.StackStreamFactory$StackFrameTraverser.consumeFrames(java.base@11.0.2/StackStreamFactory.java:534)
        at java.lang.StackStreamFactory$AbstractStackWalker.doStackWalk(java.base@11.0.2/StackStreamFactory.java:306)
        at java.lang.StackStreamFactory$AbstractStackWalker.callStackWalk(java.base@11.0.2/Native Method)
        at java.lang.StackStreamFactory$AbstractStackWalker.beginStackWalk(java.base@11.0.2/StackStreamFactory.java:370)
        at java.lang.StackStreamFactory$AbstractStackWalker.walk(java.base@11.0.2/StackStreamFactory.java:243)
        at java.lang.StackWalker.walk(java.base@11.0.2/StackWalker.java:498)
        at org.apache.logging.log4j.util.StackLocator.getCallerClass(StackLocator.java:58)
        at org.apache.logging.log4j.util.StackLocatorUtil.getCallerClass(StackLocatorUtil.java:65)
        at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
        at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
        at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
        at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:358)
        at com.linecorp.armeria.server.DefaultServiceRequestContext.newLogger(DefaultServiceRequestContext.java:208)
        at com.linecorp.armeria.server.DefaultServiceRequestContext.<init>(DefaultServiceRequestContext.java:194)
        at com.linecorp.armeria.server.DefaultServiceRequestContext.<init>(DefaultServiceRequestContext.java:127)
        at com.linecorp.armeria.server.HttpServerHandler.handleRequest(HttpServerHandler.java:351)
        at com.linecorp.armeria.server.HttpServerHandler.channelRead(HttpServerHandler.java:240)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)

Under performance testing, using jstack we can see lots of threads named with pattern armeria-common-worker-nio-* has a similar stack trace likes above. To alleviate this situation, we need to remove the multi-release mark like this to revert log4j2 to the implementation compatible with JDK8. This will lead log4j2 to use sun.reflect.Reflection.getCallerClass and leave a WARNING log as follows when the service is start. But the performance is way better than with the log4j2 for the JDK11.

WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.

My testing environment is:

• armeria: 0.97.0
• log4j-slf4j-impl: 2.12.1
• log4j-core: 2.12.1
• slf4j: 1.7.29
• jdk: 11.0.2
• OS: mac 10.15.1

[anuraaga]

Thanks for the report this seems like a big issue.

ServiceRequestContext.logger() has been deprecated for almost forever - time to remove it? @trustin @minwoox @ikhoon

https://github.com/line/armeria/blob/master/core/src/main/java/com/linecorp/armeria/server/ServiceRequestContext.java#L260

[trustin]

Fixed via #2341