Pubsub Subscriber initialization crashes JVM on Alpine Linux with libc6-compat

Pubsub Subscriber initialization crashes JVM on Alpine Linux with libc6-compat. Here is the stacktrace from hs_err crash log:

Stack: [0x00007fad5f2d8000,0x00007fad5f3d8aa8],  sp=0x00007fad5f3d2668,  free space=1001k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  0x0000000000025426
C  [libio_grpc_netty_shaded_netty_tcnative_linux_x86_64646374909380874916.so+0x29585]  JNI_OnLoad_netty_tcnative+0x55
C  [libjava.so+0xda62]  Java_java_lang_ClassLoader_00024NativeLibrary_load+0xf2

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  java.lang.ClassLoader$NativeLibrary.load(Ljava/lang/String;Z)V+0
j  java.lang.ClassLoader.loadLibrary0(Ljava/lang/Class;Ljava/io/File;)Z+328
j  java.lang.ClassLoader.loadLibrary(Ljava/lang/Class;Ljava/lang/String;Z)V+48
j  java.lang.Runtime.load0(Ljava/lang/Class;Ljava/lang/String;)V+57
j  java.lang.System.load(Ljava/lang/String;)V+7
j  io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryUtil.loadLibrary(Ljava/lang/String;Z)V+5
v  ~StubRoutines::call_stub
J 1656  sun.reflect.NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (0 bytes) @ 0x00007fad4ec3aa37 [0x00007fad4ec3a9c0+0x77]
J 1655 C1 sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (104 bytes) @ 0x00007fad4ec42554 [0x00007fad4ec41380+0x11d4]
J 587 C1 sun.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (10 bytes) @ 0x00007fad4e8cfe0c [0x00007fad4e8cfd00+0x10c]
J 586 C1 java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (62 bytes) @ 0x00007fad4e8d05dc [0x00007fad4e8d01e0+0x3fc]
j  io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader$1.run()Ljava/lang/Object;+53
v  ~StubRoutines::call_stub
J 697  java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object; (0 bytes) @ 0x00007fad4e9597cf [0x00007fad4e959780+0x4f]
j  io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(Ljava/lang/Class;Ljava/lang/String;Z)V+10
j  io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibrary(Ljava/lang/ClassLoader;Ljava/lang/String;Z)V+15
j  io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.load(Ljava/lang/String;Ljava/lang/ClassLoader;)V+402
j  io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(Ljava/lang/ClassLoader;[Ljava/lang/String;)V+33
j  io.grpc.netty.shaded.io.netty.handler.ssl.OpenSsl.loadTcNative()V+173
j  io.grpc.netty.shaded.io.netty.handler.ssl.OpenSsl.<clinit>()V+152
v  ~StubRoutines::call_stub
j  io.grpc.netty.shaded.io.grpc.netty.GrpcSslContexts.defaultSslProvider()Lio/grpc/netty/shaded/io/netty/handler/ssl/SslProvider;+0
j  io.grpc.netty.shaded.io.grpc.netty.GrpcSslContexts.configure(Lio/grpc/netty/shaded/io/netty/handler/ssl/SslContextBuilder;)Lio/grpc/netty/shaded/io/netty/handler/ssl/SslContextBuilder;+1
j  io.grpc.netty.shaded.io.grpc.netty.GrpcSslContexts.forClient()Lio/grpc/netty/shaded/io/netty/handler/ssl/SslContextBuilder;+3
j  io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder$NettyTransportFactory$DefaultNettyTransportCreationParamsFilterFactory.<init>(Lio/grpc/netty/shaded/io/grpc/netty/NettyChannelBuilder$NettyTranspo
rtFactory;Lio/grpc/netty/shaded/io/netty/handler/ssl/SslContext;)V+23
j  io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder$NettyTransportFactory$DefaultNettyTransportCreationParamsFilterFactory.<init>(Lio/grpc/netty/shaded/io/grpc/netty/NettyChannelBuilder$NettyTranspo
rtFactory;Lio/grpc/netty/shaded/io/netty/handler/ssl/SslContext;Lio/grpc/netty/shaded/io/grpc/netty/NettyChannelBuilder$1;)V+3
j  io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder$NettyTransportFactory.<init>(Lio/grpc/netty/shaded/io/grpc/netty/NettyChannelBuilder$TransportCreationParamsFilterFactory;Ljava/lang/Class;Ljava/u
til/Map;Lio/grpc/netty/shaded/io/grpc/netty/NegotiationType;Lio/grpc/netty/shaded/io/netty/handler/ssl/SslContext;Lio/grpc/netty/shaded/io/netty/channel/EventLoopGroup;IIIJJZLio/grpc/internal/TransportTra
cer;)V+45
j  io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder.buildTransportFactory()Lio/grpc/internal/ClientTransportFactory;+59
j  io.grpc.internal.AbstractManagedChannelImplBuilder.build()Lio/grpc/ManagedChannel;+10
j  com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel()Lio/grpc/ManagedChannel;+261
j  com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel()Lcom/google/api/gax/rpc/TransportChannel;+19
j  com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel()Lcom/google/api/gax/rpc/TransportChannel;+52
j  com.google.cloud.pubsub.v1.Subscriber.doStart()V+25
j  com.google.api.core.AbstractApiService$InnerService.doStart()V+4
j  com.google.common.util.concurrent.AbstractService.startAsync()Lcom/google/common/util/concurrent/Service;+33
j  com.google.api.core.AbstractApiService.startAsync()Lcom/google/api/core/ApiService;+4
j  com.google.cloud.pubsub.v1.Subscriber.startAsync()Lcom/google/api/core/ApiService;+1

[chrisribble]

@ajaaym the issue you linked to suggests installing libc-compat, but this issue clearly states that the issue happens with libc-compat installed.

It’s been a while since we moved away from Alpine, but IIRC the issue is that the netty tcnative needs to be built specially for Alpine’s SSL implementation (i.e. https://github.com/pires/netty-tcnative-alpine) and gRPC is pulling in a netty-tcnative implementation that uses JNI against the symbols in OpenSSL in a way that doesn’t work on Alpine.

I played around with trying to get that netty-tcnative-alpine project to build and figured I would force the dependency resolution to use my built version instead of the version of netty-tcnative that was specified by the gRPC pom, but I couldn’t get the netty-tcnative-alpine build to work so I gave up and switched to grc distroless (as noted above).

[sduskis]

We believe that the problem relates to grpc and SSL. You can see some information about the issue here. I haven’t tried it myself, but as best I can tell, you would have to exclude io.grpc:grpc-netty-shaded, include io.grpc:grpc-netty and include netty-tcnative-alpine.