JVM Crash while starting

So we’ve been using the Datadog Agent for more than 12 months and everything seemed ok. We’ve recently added a wrapper around our cron jobs and we’re able to pick ocasional JVM crashes when starting these jobs.

We used to have dd-trace-java 0.8 but we tried upgrading to the latest 0.18 and the crashes still occurred. We disabled the agent for a few days, to test if this could be the issue and the crashes disappeared. There seems to be a connection. The other variables where not changed (no new deploy, same machine, etc).

Attached is a simple log of the JVM starting. Do you have any similar cases? We found the crash might be related to bytebuddy which is used in this agent.

java -Djava.awt.headless=true -Xms128m -Xmx1024m -XX:+UseConcMarkSweepGC -Dlog4j.configuration=file:/folder/utils.properties -javaagent:/folder/shared/dd-java-agent.jar -jar /folder/current --config /folder/server.properties job-name --threads=1 

[main] INFO datadog.trace.agent.ot.DDTraceOTInfo - dd-trace - version: 0.18.0~d1e8ed12
[main] INFO datadog.trace.agent.ot.DDTracer - New instance: DDTracer-4416d64f{ serviceName=utils, writer=DDAgentWriter { api=DDApi { tracesEndpoint=http://localhost:8126/v0.4/traces } }, sampler=AllSampler { sample=true }, runtimeId=98ce9072, defaultSpanTags={}}
[main] INFO datadog.trace.agent.tooling.VersionLogger - dd-trace-ot - version: 0.18.0~d1e8ed12
[main] INFO datadog.trace.agent.tooling.VersionLogger - dd-trace-api - version: 0.18.0~d1e8ed12
[main] INFO datadog.trace.agent.tooling.VersionLogger - dd-java-agent - version: 0.18.0~d1e8ed12
[main] INFO datadog.trace.agent.jmxfetch.JMXFetch - JMXFetch is disabled

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fdc3f3ab9f2, pid=20229, tid=0x00007fdbdbbfb700
#
# JRE version: OpenJDK Runtime Environment (8.0_181-b13) (build 1.8.0_181-8u181-b13-2~deb9u1-b13)
# Java VM: OpenJDK 64-Bit Server VM (25.181-b13 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x7b89f2]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/user/hs_err_pid20229.log
#
# Compiler replay data is saved as:
# /home/user/replay_pid20229.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
/folder/job.sh: line 47: 20216 Aborted               

Bytebuddy class:

C2: 5798 1763 4 net.bytebuddy.description.type.TypeDescription$Generic$OfTypeVariable::accept (8 bytes)

[raphw]

This is kind of difficult as the program has no real control over how the JVM is translating a program to machine code. Any program might run into this bug if only the profile is set right. You can of course influence how C2 handles code (or even turn it of via -Xint or -client).

If I remember correctly, you can even turn off compilation for specific methods only. You can probably find a lot in this flag explorer and try out a few things: https://chriswhocodes.com/hotspot_options_jdk7.html

[knokio]

It was solved by upgrading the JRE