Linux: Native Crash lwjgl2 OpenJDK 11.0.5

So, originally this has been discovered by @jayfella but I don’t think we’ve been tracking this officially already, thus:

Just trying to run the most simple example (here app.start()) using lwjgl2 fails as follows:

Jan. 11, 2020 3:11:12 NACHM. com.jme3.system.JmeDesktopSystem initialize
INFORMATION: Running on jMonkeyEngine 3.3.0-beta1
 * Branch: HEAD
 * Git Hash: bd1b6d2
 * Build Date: 2019-12-22
Inconsistency detected by ld.so: dl-lookup.c: 111: check_match: Assertion `version->filename == NULL || ! _dl_name_match_p (version->filename, map)' failed!

However opposing to the claim this should be working with 11.0.4, I’ve seen reports about 11.0.3 here.

Either way this is unfortunate as 11.0.5 is the one available and I think will replace Java 8 soon. I am not sure if we can do anything about it, but it should be tracked so people who encounter this know it.

Edit: https://github.com/lattera/glibc/blob/master/elf/dl-lookup.c#L111 this is why it fails, and I am uncertain how openJDK could break that other than by changing the way they invoke glibc to load symbols.

Can anyone manage to find a tracking bug for openJDK? Maybe they don’t know it as everyone is just downgrading to openJDK8?

[MeFisto94]

@stephengold @jayfella @pspeed42 @riccardobl and who else it may concern:

My Investigation is complete, so here’s a “quick” wrap-up for you guys: LWJGL2 crashes on Ubuntu’s JVM 9-11+ only. It is a native crash. Installing any other JVM (e.g. AdoptOpenJDK) works. Due to Ubuntu’s popularity it’s actually a rather urgent issue.

Technical Details

LWJGL2 was compiled against Java 8 or lower JNI Libraries. It links only against `libjawt`, which contains the code to load up AWT. In Java <= 8, libjawt's methods were exported with a version symbol `SUNW_private1.1` (or similar). This leads to LWJGL2 linking against this version symbol as required/preferred version. With Java 9, they dropped this symbol as it had only a legacy purpose because of lacking compiler flags about symbol visibility (not exporting/exporting symbols).

This is where this story mostly ends for the OpenJDK Guys. Their recommendation is to just recompile LWJGL2 against a more recent java.

Now why does this only affect Ubuntu and why does the jvm crash instead of throw, say, an UnsatisfiedLinkException? The reason is that this is coupled with another bug: On AdoptOpenJDK the missing version symbol is ignored and the method is still loaded. On Ubuntu, libc (the “implementation” of the C Language) asserts with the above message.

Why? For AdoptOpenJDK, Debian, etc. the libjawt.so still contains plenty of Version Symbols, namely GLIBC_2.5.5 and others. Ubuntu seems to have a different approach and strips this symbols to have smaller files.

This means that only on Ubuntu, libjawt.so doesn’t have the version symbol table at all, which is why I suspect that libc asserts. It tries to load a versioned dependency from a file not having any version symbols at all.

I’ve reported this to the libc guys in February but didn’t receive any solution since then. I did poke the guys once and basically the response was that they didn’t have the time/motivation yet to extract the Ubuntu JVM and lwjgl to inspect the behavior and that I should provide a self-contained test case.

When doing so, I could debug the problem myself. But I didn’t want to go to the hassle and re-create the jvm’s call stack, building libc in debug mode or whatever would’ve been required for this. I’ve also figured that a patch to libc would take at least a year until this is fixed on Ubuntu.

Furthermore what we’re doing there is “borderline unsupported” anyway and the fix to delete the SUNWprivate label would be very trivial.

Thus, I’ve forked lwjgl2 at https://github.com/MeFisto94/lwjgl and made it build with Github Actions (I’ve taken the post Java8 patches from Debian).

My Proposal is to try to merge the existing lwjgl2 jar with the .so created with my fork and try to see if that works on Ubuntu as expected and then switch the engine’s lwjgl2 dependency to this repository. Details like if we should provide org.jmonkeyengine.lwjgl2 or something can be discussed later.

Also before recommending to just delete lwjgl2, please consider:

• Using LWJGL3 for the SDK would be illegal (specs require to run on the Main Thread) and will crash on Mac OS X. This is what happened to jay’s plugins/editor and what will happen to all IntelliJ Plugins or whatever are around. Furthermore, this will affect ALL guys writing their own kind-of-launchers as well as using something like launch4j, probably even the JavaFX packaging.

• Using LWJGL3 on Mac OS X for anything else is problematic (one needs to pass -XStartOnMainThread=true to the JVM, because native threads != the java main thread, otherwise).

• While lwjgl2 may be old and have some messy code/bugs, lwjgl3 changed a lot of things which make apps work unreliable. Specifically text input on non US layouts is “broken” or bad design.

• The heavy lifting has already done by me, so it’s just about thorough testing and packaging.

[Sailsman63]

^ My understanding was that it boiled down to This glibc bug. Situations when the runtime tries to load multiple native libraries in quick succession (Such as lwjgl2, or older versions of jogamp JOGL) can lead to data races.

The JREs that show the issue seem to have statically linked a version of glibc that has the bug., whereas those that do not are linking a different version of glibc. The newer versions of OpenJDK/Oracle builds seem to catch the data race and error out rather than actually link a fixed version of glibc.