Segfault on Fedora 32, Java 15, OpenJDK 20.9

Description

My Teku node segfaulted while syncing from genesis.

16:08:57.993 INFO  - Sync Event  *** Current slot: 645644, Head slot: 47776, Connected peers: 28
16:09:10.817 INFO  - Sync Event  *** Current slot: 645645, Head slot: 47808, Connected peers: 27
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fa160cb3b31, pid=210481, tid=211314
#
# JRE version: OpenJDK Runtime Environment 20.9 (15.0+36) (build 15+36)
# Java VM: OpenJDK 64-Bit Server VM 20.9 (15+36, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0xd8fb31]  ObjectSynchronizer::get_lock_owner(ThreadsList*, Handle)+0x51
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h" (or dumping to /home/michael/teku-medalla/core.210481)
#
# An error report file with more information is saved as:
# /home/michael/teku-medalla/hs_err_pid210481.log
Compiled method (n/a) 12689402 20398     n 0       sun.management.ThreadImpl::getThreadInfo1 (native)
 total in heap  [0x00007fa148ba4a90,0x00007fa148ba4e18] = 904
 relocation     [0x00007fa148ba4be8,0x00007fa148ba4c20] = 56
 main code      [0x00007fa148ba4c20,0x00007fa148ba4e18] = 504
#
# If you would like to submit a bug report, please visit:
#   https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora&component=java-latest-openjdk&version=32
#
./teku.sh: line 9: 210481 Aborted                 (core dumped) ./teku-0.12.12/bin/teku --network=medalla --rest-api-enabled --rest-api-docs-enabled --metrics-enabled

Steps to Reproduce

Unsure, looks like some kind of race condition, or possibly a bug in the JVM. If it happens again I’ll likely try a different JVM version.

I was running with these arguments:

#!/bin/bash

export JAVA_OPTS="-Xmx2g"

./teku-0.12.12/bin/teku \
    --network=medalla \
    --rest-api-enabled \
    --rest-api-docs-enabled \
    --metrics-enabled

Versions

• Software version: teku/v0.12.12/linux-x86_64/-redhatinc-openjdk64bitservervm-java-15
• Java version:

openjdk version "15" 2020-09-15
OpenJDK Runtime Environment 20.9 (build 15+36)
OpenJDK 64-Bit Server VM 20.9 (build 15+36, mixed mode, sharing)

• OS Name & Version: Fedora release 32 (Thirty Two)
• Kernel Version: Linux my-hostname 5.8.15-201.fc32.x86_64 #1 SMP Thu Oct 15 15:56:44 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
• Virtual Machine software & version: N/A
• Docker Version: N/A
• Cloud VM, type, size: N/A
• CPU: Core i3 530 (2010)

Additional Information

This issue looks similar: https://github.com/ConsenSys/teku/issues/3079

[ajsutton]

This hasn’t been an ongoing issue and JDK versions have progressed a lot since. Given it only occurred when teku wouldn’t be able to continue due to lack of memory anyway I’m going to close this.

[ajsutton]

Copying my comment from #3079 here: https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8249871 looks extremely similar - the ThreadLocal.getMap method is called by ThreadLocal.remove and is also small enough to have likely been inlined. Unfortunately its still unresolved.

All of these reports are in an area which is accessing thread information in some form which points back to that same bug.