Can't run CrateDB from development sandbox on JDK <= 14

Hi there,

similar to what @BaurzhanSakhariev reported at https://github.com/crate/crate/pull/10978#issuecomment-782029968, I am getting this stacktrace when trying to invoke CrateDB like outlined at [1] from the root directory of the working tree.

./gradlew clean app:run

java.lang.ExceptionInInitializerError

> Task :app:run FAILED
Exception in thread "main" java.lang.ExceptionInInitializerError
        at java.base/jdk.internal.icu.text.NormalizerBase$NFKDMode.getNormalizer2(NormalizerBase.java:305)
        at java.base/jdk.internal.icu.text.NormalizerBase.normalize(NormalizerBase.java:457)
        at java.base/jdk.internal.icu.text.NormalizerBase.normalize(NormalizerBase.java:461)
        at java.base/java.text.Normalizer.normalize(Normalizer.java:159)
        at java.base/sun.security.x509.AVA.toRFC2253CanonicalString(AVA.java:987)
        at java.base/sun.security.x509.RDN.toRFC2253StringInternal(RDN.java:437)
        at java.base/sun.security.x509.RDN.toRFC2253String(RDN.java:417)
        at java.base/sun.security.x509.X500Name.getRFC2253CanonicalName(X500Name.java:724)
        at java.base/sun.security.x509.X500Name.equals(X500Name.java:416)
        at java.base/sun.security.pkcs.PKCS7.getCertificate(PKCS7.java:694)
        at java.base/sun.security.pkcs.SignerInfo.getCertificate(SignerInfo.java:253)
        at java.base/sun.security.pkcs.SignerInfo.verify(SignerInfo.java:401)
        at java.base/sun.security.pkcs.PKCS7.verify(PKCS7.java:578)
        at java.base/sun.security.pkcs.PKCS7.verify(PKCS7.java:595)
        at java.base/sun.security.pkcs.SignerInfo.getTimestamp(SignerInfo.java:566)
        at java.base/sun.security.pkcs.SignerInfo.verify(SignerInfo.java:324)
        at java.base/sun.security.pkcs.PKCS7.verify(PKCS7.java:578)
        at java.base/sun.security.pkcs.PKCS7.verify(PKCS7.java:595)
        at java.base/sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:293)
        at java.base/sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:269)
        at java.base/java.util.jar.JarVerifier.processEntry(JarVerifier.java:316)
        at java.base/java.util.jar.JarVerifier.update(JarVerifier.java:230)
        at java.base/java.util.jar.JarFile.initializeVerifier(JarFile.java:749)
        at java.base/java.util.jar.JarFile.ensureInitialization(JarFile.java:1016)
        at java.base/java.util.jar.JavaUtilJarAccessImpl.ensureInitialization(JavaUtilJarAccessImpl.java:72)
        at java.base/jdk.internal.loader.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:873)
        at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:811)
        at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:723)
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:646)
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:604)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:168)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        at io.crate.types.DataTypeXContentExtension.getXContentWriters(DataTypeXContentExtension.java:40)
        at org.elasticsearch.common.xcontent.XContentBuilder.<clinit>(XContentBuilder.java:119)
        at org.elasticsearch.common.settings.Setting.arrayToParsableString(Setting.java:1363)
        at org.elasticsearch.common.settings.Setting$ListSetting.lambda$new$0(Setting.java:1388)
        at org.elasticsearch.common.settings.Setting.<init>(Setting.java:152)
        at org.elasticsearch.common.settings.Setting$ListSetting.<init>(Setting.java:1385)
        at org.elasticsearch.common.settings.Setting.listSetting(Setting.java:1337)
        at org.elasticsearch.common.settings.Setting.listSetting(Setting.java:1305)
        at org.elasticsearch.env.Environment.<clinit>(Environment.java:55)
        at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:66)
        at io.crate.bootstrap.CrateDB.createEnv(CrateDB.java:112)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:81)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
        at org.elasticsearch.cli.Command.main(Command.java:90)
        at io.crate.bootstrap.CrateDB.main(CrateDB.java:91)
        at io.crate.bootstrap.CrateDB.main(CrateDB.java:84)
Caused by: java.lang.NullPointerException: Cannot invoke "java.io.InputStream.available()" because "is" is null
        at java.base/jdk.internal.icu.impl.ICUBinary.getRequiredData(ICUBinary.java:94)
        at java.base/jdk.internal.icu.impl.NormalizerImpl.load(NormalizerImpl.java:431)
        at java.base/jdk.internal.icu.impl.Norm2AllModes$Norm2AllModesSingleton.<init>(Norm2AllModes.java:274)
        at java.base/jdk.internal.icu.impl.Norm2AllModes$NFKCSingleton.<clinit>(Norm2AllModes.java:290)
        at java.base/jdk.internal.icu.impl.Norm2AllModes.getNFKCInstance(Norm2AllModes.java:263)
        at java.base/jdk.internal.icu.text.Normalizer2.getNFKDInstance(Normalizer2.java:123)
        at java.base/jdk.internal.icu.text.NormalizerBase$NFKDModeImpl.<clinit>(NormalizerBase.java:181)
        ... 48 more

I’ve also attached some information about the macOS environment I am working in. The mitigation is probably very easy, I am just not seeing it.

Thank you already for looking into this.

With kind regards, Andreas.

[1] https://github.com/crate/crate/blob/4.4.2/devs/docs/basics.rst#manual-build

---

> Task :server:getVersion
gitTag:         4.4.0-133-gf8b0533c02
version:        4.5.0-SNAPSHOT-f8b0533c02
buildDate:      202103042106
buildShortHash: f8b0533c02

$ java -version
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-46)
OpenJDK 64-Bit Server VM (build 14.0.2+12-46, mixed mode, sharing)

$ uname -a
Darwin sink.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64

$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.15.7
BuildVersion:	19H2

[mfussenegger]

Closing this - can’t reproduce it on Linux and there is a known workaround (update java).

If you feel like it you could create a PR to update the docs.

[amotl]

Thank you for following up on this. Unfortunately, I still observe the same behavior on macOS, right. It still happens both on Java 11 and Java 14, I am on 2c67256 and invoke ./gradlew clean app:run.

Hereby, I am sharing my observations with you because the origin of the exception trace slightly changed. Apparently, it is always something about the ICU library being involved to process the path name when addressing the filesystem in any way.

1. On the first occasion, the classloader seems to have been involved to some degree when accessing the filesystem for verifying Jar package signatures, see https://github.com/crate/crate/issues/11102#issue-822473861.

2. Now, it is apparently org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetadata accessing the filesystem by going to java.nio.file.Files.newDirectoryStream, in turn going to jdk.internal.icu for concluding path name matches.

java.lang.ExceptionInInitializerError, Part 2

[2021-05-04T19:47:18,391][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [Griesmauerkogel] fatal error in thread [main], exiting
java.lang.ExceptionInInitializerError: null
        at jdk.internal.icu.text.NormalizerBase$NFCMode.getNormalizer2(NormalizerBase.java:313) ~[?:?]
        at jdk.internal.icu.text.NormalizerBase.normalize(NormalizerBase.java:457) ~[?:?]
        at jdk.internal.icu.text.NormalizerBase.normalize(NormalizerBase.java:461) ~[?:?]
        at java.text.Normalizer.normalize(Normalizer.java:159) ~[?:?]
        at java.util.regex.Pattern.normalizeClazz(Pattern.java:1541) ~[?:?]
        at java.util.regex.Pattern.normalize(Pattern.java:1472) ~[?:?]
        at java.util.regex.Pattern.compile(Pattern.java:1745) ~[?:?]
        at java.util.regex.Pattern.<init>(Pattern.java:1430) ~[?:?]
        at java.util.regex.Pattern.compile(Pattern.java:1095) ~[?:?]
        at sun.nio.fs.MacOSXFileSystem.compilePathMatchPattern(MacOSXFileSystem.java:47) ~[?:?]
        at sun.nio.fs.UnixFileSystem.getPathMatcher(UnixFileSystem.java:300) ~[?:?]
        at java.nio.file.Files.newDirectoryStream(Files.java:537) ~[?:?]
        at org.elasticsearch.gateway.MetadataStateFormat.findMaxGenerationId(MetadataStateFormat.java:356) ~[crate-server.jar:?]
        at org.elasticsearch.gateway.MetadataStateFormat.loadLatestStateWithGeneration(MetadataStateFormat.java:435) ~[crate-server.jar:?]
        at org.elasticsearch.gateway.MetadataStateFormat.loadLatestState(MetadataStateFormat.java:459) ~[crate-server.jar:?]
        at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetadata(NodeEnvironment.java:406) ~[crate-server.jar:?]
        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:302) ~[crate-server.jar:?]
        at org.elasticsearch.node.Node.<init>(Node.java:257) ~[crate-server.jar:?]
        at io.crate.node.CrateNode.<init>(CrateNode.java:54) ~[main/:?]
        at org.elasticsearch.bootstrap.BootstrapProxy$1.<init>(BootstrapProxy.java:193) ~[main/:?]
        at org.elasticsearch.bootstrap.BootstrapProxy.setup(BootstrapProxy.java:193) ~[main/:?]
        at org.elasticsearch.bootstrap.BootstrapProxy.init(BootstrapProxy.java:267) ~[main/:?]
        at io.crate.bootstrap.CrateDB.init(CrateDB.java:158) ~[main/:?]
        at io.crate.bootstrap.CrateDB.execute(CrateDB.java:138) ~[main/:?]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:81) ~[crate-server.jar:?]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[crate-cli.jar:?]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[crate-cli.jar:?]
        at io.crate.bootstrap.CrateDB.main(CrateDB.java:91) ~[main/:?]
        at io.crate.bootstrap.CrateDB.main(CrateDB.java:84) ~[main/:?]
Caused by: java.lang.NullPointerException: Cannot invoke "java.io.InputStream.available()" because "is" is null
        at jdk.internal.icu.impl.ICUBinary.getRequiredData(ICUBinary.java:94) ~[?:?]
        at jdk.internal.icu.impl.NormalizerImpl.load(NormalizerImpl.java:431) ~[?:?]
        at jdk.internal.icu.impl.Norm2AllModes$Norm2AllModesSingleton.<init>(Norm2AllModes.java:274) ~[?:?]
        at jdk.internal.icu.impl.Norm2AllModes$NFCSingleton.<clinit>(Norm2AllModes.java:286) ~[?:?]
        at jdk.internal.icu.impl.Norm2AllModes.getNFCInstance(Norm2AllModes.java:259) ~[?:?]
        at jdk.internal.icu.text.Normalizer2.getNFCInstance(Normalizer2.java:90) ~[?:?]
        at jdk.internal.icu.text.NormalizerBase$NFCModeImpl.<clinit>(NormalizerBase.java:185) ~[?:?]
        ... 29 more

Maybe you can figure out that we would have to turn more things into runtime components, similar to #11307, or apply different countermeasures?

Maybe MacOS specific?

Maybe. There is indeed a thing specific to macOS involved here:

java.util.regex.Pattern.compile(Pattern.java:1095) ~[?:?]
sun.nio.fs.MacOSXFileSystem.compilePathMatchPattern(MacOSXFileSystem.java:47) ~[?:?]
sun.nio.fs.UnixFileSystem.getPathMatcher(UnixFileSystem.java:300) ~[?:?]
java.nio.file.Files.newDirectoryStream(Files.java:537) ~[?:?]