grpc-java libio_grpc_netty_shaded_netty_transport_native_epoll_x86_64 jvm crash after update to 1.45.1

I’m having a weird issue when I execute an junit integration test after I upgrade grpc versions. It is getting a JVM crash on Linux only, while executing the same test case on Windows poses no issue. And I do not understand the reason. Theoretically, I would expect the same behavior. It is the same mvnw clean verify -Djacoco.skip=false cmd on both OS’es. Attached is the hs file.

At runtime it does not seem to pose any issue on a Linux OS. The junit is using ‘grpc inProcess for both client/server’.

When using gpc version 1.40.1, my test passes successfully

[INFO] |  +- io.opentracing.contrib:opentracing-grpc:jar:0.2.3:compile
[INFO] |  +- net.devh:grpc-spring-boot-starter:jar:2.13.0.RELEASE:compile
[INFO] |  |  \- net.devh:grpc-server-spring-boot-starter:jar:2.13.0.RELEASE:compile
[INFO] |  |     \- net.devh:grpc-server-spring-boot-autoconfigure:jar:2.13.0.RELEASE:compile
[INFO] |  |        \- io.grpc:grpc-services:jar:1.42.1:compile
[INFO] |  +- io.grpc:grpc-protobuf:jar:1.43.2:compile
[INFO] |  |  +- com.google.api.grpc:proto-google-common-protos:jar:2.0.1:compile
[INFO] |  |  \- io.grpc:grpc-protobuf-lite:jar:1.43.2:compile
[INFO] |  +- io.grpc:grpc-stub:jar:1.43.2:compile
[INFO] +- net.devh:grpc-client-spring-boot-starter:jar:2.13.0.RELEASE:compile
[INFO] |  \- net.devh:grpc-client-spring-boot-autoconfigure:jar:2.13.0.RELEASE:compile
[INFO] |     \- net.devh:grpc-common-spring-boot:jar:2.13.0.RELEASE:compile
[INFO] +- io.grpc:grpc-netty-shaded:jar:1.40.1:compile
[INFO] +- io.grpc:grpc-core:jar:1.40.1:compile
[INFO] |  +- io.grpc:grpc-api:jar:1.40.1:compile (version selected from constraint [1.40.1,1.40.1])
[INFO] |  |  \- io.grpc:grpc-context:jar:1.40.1:compile

After upgrade: hs_err_pid1033.log

$ ./mvnw dependency:tree | grep grpc
[INFO] |  +- io.opentracing.contrib:opentracing-grpc:jar:0.2.3:compile
[INFO] |  +- net.devh:grpc-spring-boot-starter:jar:2.13.0.RELEASE:compile
[INFO] |  |  \- net.devh:grpc-server-spring-boot-starter:jar:2.13.0.RELEASE:compile
[INFO] |  |     \- net.devh:grpc-server-spring-boot-autoconfigure:jar:2.13.0.RELEASE:compile
[INFO] |  |        \- io.grpc:grpc-services:jar:1.42.1:compile
[INFO] |  +- io.grpc:grpc-protobuf:jar:1.43.2:compile
[INFO] |  |  +- com.google.api.grpc:proto-google-common-protos:jar:2.0.1:compile
[INFO] |  |  \- io.grpc:grpc-protobuf-lite:jar:1.43.2:compile
[INFO] |  +- io.grpc:grpc-stub:jar:1.43.2:compile
[INFO] +- net.devh:grpc-client-spring-boot-starter:jar:2.13.0.RELEASE:compile
[INFO] |  \- net.devh:grpc-client-spring-boot-autoconfigure:jar:2.13.0.RELEASE:compile
[INFO] |     \- net.devh:grpc-common-spring-boot:jar:2.13.0.RELEASE:compile
[INFO] +- io.grpc:grpc-netty-shaded:jar:1.45.1:compile
[INFO] +- io.grpc:grpc-core:jar:1.45.1:compile
[INFO] |  +- io.grpc:grpc-api:jar:1.45.1:compile (version selected from constraint [1.45.1,1.45.1])
[INFO] |  |  \- io.grpc:grpc-context:jar:1.45.1:compile

Any ideas of what is wrong ?

[ejona86]

With the change to Netty, gRPC will fallback to other options. If you are running on OpenJDK 8u252 or later, then it will still work. Although performance may be lower compared to a glibc system, especially on Java 8.

[ejona86]

I see. There is a choice between Nio and Epoll threads and polling, but there isn’t a way to avoid Epoll native library loading, except using grpc-netty instead of grpc-netty-shaded. io.grpc.netty.shaded.io.netty.transport.noNative gets you further because it avoids Epoll triggering the initialization of io.netty.channel.epoll.Native, as long as you don’t call any other methods within io.netty.channel.epoll.

It would be possible to delay initialization of the builder fields to avoid the epoll initialization, but it doesn’t seem to buy us too much as it would require all Alpine users to manage their own loops, lest they get a runtime crash. A crash is too horrible of a failure mode.

I think https://github.com/netty/netty/pull/12272 will be the real fix here, as it does -Wl,-z,now which disables the delayed loading of symbols. That should cause dlopen to fail and let grpc handle this gracefully by falling back to NIO. With that in place it wouldn’t be bad to avoid loading epoll when the application specifies the event loop, but the gains will be more minor at that point.