`2.13.0.RELEASE` or later crashes JVM during deployment to k8s

The context

We would like to upgrade the dependency version of net.devh:grpc-server-spring-boot-starter in our Gradle build script for our microservice.

The bug

The application does not start if using version 2.13.0.RELEASE or later. That makes the deployment fail.

Stacktrace and logs

#   https://github.com/corretto/corretto-17/issues/
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
#
# If you would like to submit a bug report, please visit:
# Core dump will be written. Default location: //core.1
#
# An error report file with more information is saved as:
# /tmp/hs_err_pid1.log
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0000000000003efe, pid=1, tid=7
#
# JRE version: OpenJDK Runtime Environment Corretto-17.0.1.12.1 (17.0.1+12) (build 17.0.1+12-LTS)
# Java VM: OpenJDK 64-Bit Server VM Corretto-17.0.1.12.1 (17.0.1+12-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, serial gc, linux-amd64)
# Problematic frame:
# C  0x0000000000003efe
#

Steps to Reproduce

It happens just right after deploying the application to our AWS QA K8S environment. Build, unit and integration tests work fine. Also running the application locally (with docker or through IntelliJ works).

The application’s environment

Which versions do you use?

• Spring (boot): 2.6.2
• grpc-spring-boot-starter: 2.13.0.RELEASE, 2.13.1.RELEASE
• kotlin: 1.6.10
• JRE version: OpenJDK Runtime Environment Corretto-17.0.1.12.1 (17.0.1+12) (build 17.0.1+12-LTS)
• Java VM: OpenJDK 64-Bit Server VM Corretto-17.0.1.12.1

Additional context With version 2.12.0.RELEASE everything works fine.

• Did it ever work before? Yes, up to version 2.12.0.RELEASE.

• Do you have a demo? No

[savbiz]

I’m not familiar with Corretto, so can you please try with a different JVM too please?

Which JVM would you prefer us to try? Would a different base Docker image, (openjdk:17-alpine for example, be fine? Right now we are using amazoncorretto:17-alpine.

OpenJDK or Eclipse Temurin would be good. This test is intended to identify whether this is specific to Corretto or the JVM in general.

Which OS are you using underneath your Corretto JRE?

Just Kubernetes (version 1.21) Platform version (AWS EKS - Amazon Elastic Kubernetes Service): eks.2

I was actually referring to the OS inside the image, but you already answered that in the first block (alpine). Please try also the plain Corretto base image amazoncorretto:17 (or similar) to rule out that this is a musl vs glibc issue. I know these are much larger, but without additional information I don’t know how to narrow it down.

Does the error also happen if you start the application locally?

No, locally it works fine starting the application on IntelliJ or local docker image ( which uses the same Dockerfile as our deployment tool does).

That is very strange. Might this be caused by differences in the config/properties?

Could you please include that log file or at least check it for relevant information?

  Unfortunately, this is hard to retrieve since it is stored on our container which is ephemeral (meaning that when the deployment fails, it gets deleted right away). We'll have to find a way to intercept this file and print it to standard output.

You might be able to do the following:

docker pull image
docker save image > image.tar
# open the tar and search for the entrypoint in the config
# append something like `|| cat /tmp/hs_err_pid1.log` to the command
docker load < image.tar
# maybe re-tag the image
docker push image

Sorry for the late reply. We have actually solved the original issue by following this workaround. It seems to be indeed a problem with Alpine images and k8 environments. We are now able to use the latest version of your library. Thanks for helping out though!

[ST-DDT]

I guess I have to ramp up my testExamples.sh once more to also test for this scenario.

@fireXtract From what I can tell, your error is not related to the original error.