JIT Compilation error with Oracle JDK 7u80

Describe the bug

When running with Oracle 7u80 JDK, after running for a while with agent enabled, the JVM process crash with the following message in the crash report:

# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (safepoint.cpp:318), pid=819438, tid=140189423261440
#  guarantee(PageArmed == 0) failed: invariant
#
# JRE version: Java(TM) SE Runtime Environment (7.0_80-b15) (build 1.7.0_80-b15)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.80-b11 mixed mode linux-amd64 compressed oops)

The stack trace is not directly related to the agent, and async-profiler is not enabled

Current thread (0x00007f806c1ba800):  VMThread [stack: 0x00007f8064b66000,0x00007f8064c66000] [id=819449]

Stack: [0x00007f8064b66000,0x00007f8064c66000],  sp=0x00007f8064c646d0,  free space=1017k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x9a320a]  VMError::report_and_die()+0x2ea
V  [libjvm.so+0x498da2]  report_vm_error(char const*, int, char const*, char const*)+0x62
V  [libjvm.so+0x8b03b4]  SafepointSynchronize::begin()+0x824
V  [libjvm.so+0x9aa96d]  VMThread::loop()+0x4bd
V  [libjvm.so+0x9aaaf0]  VMThread::run()+0x70
V  [libjvm.so+0x8238c8]  java_start(Thread*)+0x108

The last compilation events at the time of the crash indicate that co.elastic.apm.agent.servlet.ServletApiAdvice::onEnterServletService has just been compiled by the JIT compiler

Compilation events (10 events):
Event: 213.351 Thread 0x00007f806c85d000 3311   !         org.springframework.web.servlet.tags.UrlTag::replaceUriTemplateParams (215 bytes)
Event: 213.351 Thread 0x00007f806c85a800 3312   !         org.springframework.web.servlet.tags.UrlTag::createQueryString (181 bytes)
Event: 213.397 Thread 0x00007f806c85a800 nmethod 3312 0x00007f80699b2490 code [0x00007f80699b26a0, 0x00007f80699b3798]
Event: 213.427 Thread 0x00007f806c85d000 nmethod 3311 0x00007f80699b5210 code [0x00007f80699b5680, 0x00007f80699b7f58]
Event: 213.593 Thread 0x00007f806c85a800 3313             org.hsqldb.ParserDQL$CompileContext::getNextRangeVarIndex (27 bytes)
Event: 213.594 Thread 0x00007f806c85a800 nmethod 3313 0x00007f806995e690 code [0x00007f806995e7c0, 0x00007f806995e858]
Event: 213.844 Thread 0x00007f806c85d000 3314             org.hsqldb.lib.OrderedHashSet::addAll (25 bytes)
Event: 213.844 Thread 0x00007f806c85d000 nmethod 3314 0x00007f806995acd0 code [0x00007f806995ae00, 0x00007f806995ae58]
Event: 213.846 Thread 0x00007f806c85a800 3315             co.elastic.apm.agent.servlet.ServletApiAdvice::onEnterServletService (664 bytes)
Event: 213.897 Thread 0x00007f806c85a800 nmethod 3315 0x00007f80699be750 code [0x00007f80699bef20, 0x00007f80699c1fd8]

This issue is know to happen with the following conditions

• Using Oracle 7u80 JDK
• using agent 1.18.0+ up to current version on master branch
• Application uses Servlet API
• Application is under enough load to trigger JIT compilation of the Servlet Advice.

What we don’t know yet:

• If other Java 7 JDKs are affected, which might be hard or impossible to check due to no public availability of Oracle JDK after 7u80.
• If this issue can happen with other parts of the Agent or other instrumentation plugins.

Work-arounds

• Updating to a more recent JDK: either Java 8+, Oracle-supported Java 7, or Azul Java 7

• Using the -XX:CompileCommand to exclude the agent method(s) that are visible in the crash report, for the case here with onEnterServletService it would be:

-XX:CompileCommand=exclude,co/elastic/apm/agent/servlet/ServletApiAdvice.onEnterServletService

Expected behavior

• no JVM crash

Debug logs

hs_err_pid819438.log

Additional context

Similar issue was reported on our forum

When setting -XX:CompileThreshold=500 in order to try reproducing this issue faster, we have another compiler error. hs_err_pid2170058.log

[eyalkoren]

Update: a 15-hour load test with oracle-7u80-linux has finished successfully 🎉

[eyalkoren]

The -XX:CompileCommand=exclude,java.lang.invoke.LambdaForm*::* workaround proved to work also with Oracle 7u80 JDK - just ended a load test of 6 hours with 100 tps successfully.