JVM crash with grpc-java 1.42.x and alpine docker image

An attempt to upgrade from grpc-java 1.41.1 to 1.42.x ends with JVM crash. It looks like the problem is specific to Alpine Linux. It reproduces on openjdk:15-jdk-alpine and openjdk:8-alpine and goes away with a switch on openjdk:X-slim [debian] images. Maybe also affected by the fact that openjdk:X-alpine images are not maintained anymore, hence have no new JDK updates. The first version of grpc-java with the problem is 1.42.0, the versions before work fine.

It may be related to https://github.com/grpc/grpc/issues/27995

| # A fatal error has been detected by the Java Runtime Environment:   | #   | # SIGSEGV (0xb) at pc=0x0000000000003efe, pid=372, tid=0x00007fbffbc9bb10   | #   | # JRE version: OpenJDK Runtime Environment (8.0_212-b04) (build 1.8.0_212-b04)   | # Java VM: OpenJDK 64-Bit Server VM (25.212-b04 mixed mode linux-amd64 compressed oops)   | # Derivative: IcedTea 3.12.0   | # Distribution: Custom build (Sat May 4 17:33:35 UTC 2019)   | # Problematic frame:   | # C 0x0000000000003efe   | #   | # Core dump written. Default location: /temporal-java-client/temporal-kotlin/core or core.372   | #   | # An error report file with more information is saved as:   | # /temporal-java-client/temporal-kotlin/hs_err_pid372.log   | #   | # If you would like to submit a bug report, please include   | # instructions on how to reproduce the bug and visit:   | # https://icedtea.classpath.org/bugzilla   | #

hs_err_pid372.log

[artemptushkin]

I had the same issue having grpc libraries in the classpath like:

io.grpc:grpc-stub:1.42.1

and apline:3.15.0 with JDK

openjdk version "11.0.13" 2021-10-19
OpenJDK Runtime Environment (build 11.0.13+8-alpine-r0)
OpenJDK 64-Bit Server VM (build 11.0.13+8-alpine-r0, mixed mode)

Thanks to @ejona86, what I’ve made:

• I added gcompat lib

apk add gcompat

• Run Java like this now

LD_PRELOAD=/lib/libgcompat.so.0 java -cp @/app/jib-classpath-file io.my.MyApplication

Or Dockerfile should include just:

RUN apk add gcompat

ENV LD_PRELOAD=/lib/libgcompat.so.0

[ejona86]

TL;DR: Alpine doesn’t have compatibility for the __strndup symbol. I don’t know why the behavior is k8s-dependent, though. And it’ll take some more research to determine appropriate next steps.

Looking at objdump output, it looks like the problem is happening in parsePackagePrefix.

libio_grpc_netty_shaded_netty_transport_native_epoll_x86_642526953976876250345.so+0xb487:

    b47c:	48 89 de             	mov    %rbx,%rsi
    b47f:	4c 89 e7             	mov    %r12,%rdi
    b482:	e8 59 fe ff ff       	call   b2e0 <parsePackagePrefix>
    b487:	83 7d dc ff          	cmpl   $0xffffffff,-0x24(%rbp)

But I don’t see any obvious places the stack could get corrupted in parsePackagePrefix and there’s no callout to a passed function. I’m suspicious though that the linker is broken. This is in parsePackagePrefix:

    b395:	e8 5e 8b ff ff       	call   3ef8 <__strndup@plt>

The address displayed is relative to the .so, so isn’t the problem itself. It jumps to the PLT:

0000000000003ef8 <__strndup@plt>:
    3ef8:       ff 25 52 c2 20 00       jmp    *0x20c252(%rip)        # 210150 <__strndup@GLIBC_2.2.5>
    3efe:       68 36 00 00 00          push   $0x36
    3f03:       e9 80 fc ff ff          jmp    3b88 <.plt>

And the indirect jump goes to the GOT which should be filled with the adjusted address of 3efe. But maybe something is broken in the linker and it didn’t get adjusted?

/ # apk add gcompat
/ # ldd libnetty_transport_native_epoll_x86_64-4.1.63.so 
	ldd (0x7fa4cfc2f000)
	librt.so.1 => ldd (0x7fa4cfc2f000)
	libdl.so.2 => ldd (0x7fa4cfc2f000)
	libc.so.6 => ldd (0x7fa4cfc2f000)
Error relocating libnetty_transport_native_epoll_x86_64-4.1.63.so: __strndup: symbol not found

Well, there we go… Older versions of epoll linked against strndup, not __strndup. This difference may have been caused by a glibc upgrade when compiling.