TensorFlow library not loading with GraalVM Native Image

Description

I am working on an implementation of the covid19-detection example code, but using Quarkus to serve the model and also support GraalVM Native Image.

The project is located here: https://github.com/murphye/djl-demo/tree/master/covid19-detection-quarkus

The application runs fine on the JVM, but when running in native mode, the TensorFlow libraries are not being loaded (i.e. System.loadLibrary).

Important: I cannot find a reference in the DJL code for System.loadLibrary and I do not understand how the TensorFlow libraries are actually loaded. If I better understood how the mechanism worked, I could better diagnose it. It does seem to be related to Bytedeco which I am not familiar with.

Here is the code that I am running to demonstrate the issue:

        LibUtils.loadLibrary(); // Forcing the library to load to demo error
        System.out.println("Library Path: " + System.getProperty("org.bytedeco.javacpp.platform.preloadpath"));

        // See if TF loaded correctly or not. If not, expect java.lang.UnsatisfiedLinkError
        TfEngine.getInstance().debugEnvironment();

Error Message

Here is the output of the error when running this code. The TF libraries are downloaded and placed in /Users/ermurphy/.tensorflow/cache/2.1.0-a-SNAPSHOT-cpu-osx-x86_64 correctly.

__  ____  __  _____   ___  __ ____  ______ 
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
2020-05-11 19:22:10,136 INFO  [io.quarkus] (main) covid19-detection-quarkus 1.0-SNAPSHOT (powered by Quarkus 1.4.2.Final) started in 0.056s. Listening on: http://0.0.0.0:8080
2020-05-11 19:22:10,137 INFO  [io.quarkus] (main) Profile prod activated. 
2020-05-11 19:22:10,137 INFO  [io.quarkus] (main) Installed features: [cdi, resteasy, resteasy-jackson]
2020-05-11 19:22:23,584 INFO  [ai.djl.ten.eng.LibUtils] (executor-thread-1) Downloading libstdc++.6.dylib ...
2020-05-11 19:22:24,752 INFO  [ai.djl.ten.eng.LibUtils] (executor-thread-1) Downloading libjnitensorflow.dylib ...
2020-05-11 19:22:24,987 INFO  [ai.djl.ten.eng.LibUtils] (executor-thread-1) Downloading libgcc_s.1.dylib ...
2020-05-11 19:22:25,203 INFO  [ai.djl.ten.eng.LibUtils] (executor-thread-1) Downloading THIRD_PARTY_TF_JNI_LICENSES ...
2020-05-11 19:22:25,448 INFO  [ai.djl.ten.eng.LibUtils] (executor-thread-1) Downloading libtensorflow.2.dylib ...
2020-05-11 19:22:37,506 INFO  [ai.djl.ten.eng.LibUtils] (executor-thread-1) Downloading libjnimklml.dylib ...
2020-05-11 19:22:37,656 INFO  [ai.djl.ten.eng.LibUtils] (executor-thread-1) Downloading libiomp5.dylib ...
2020-05-11 19:22:38,182 INFO  [ai.djl.ten.eng.LibUtils] (executor-thread-1) Downloading libmkldnn.0.dylib ...
2020-05-11 19:22:38,934 INFO  [ai.djl.ten.eng.LibUtils] (executor-thread-1) Downloading LICENSE ...
2020-05-11 19:22:39,038 INFO  [ai.djl.ten.eng.LibUtils] (executor-thread-1) Downloading libmklml.dylib ...
2020-05-11 19:22:42,655 INFO  [ai.djl.ten.eng.LibUtils] (executor-thread-1) Downloading libjnimkldnn.dylib ...
2020-05-11 19:22:42,833 INFO  [ai.djl.ten.eng.LibUtils] (executor-thread-1) Downloading libgomp.1.dylib ...
Library Path: /Users/ermurphy/.tensorflow/cache/2.1.0-a-SNAPSHOT-cpu-osx-x86_64
2020-05-11 19:22:42,979 INFO  [ai.djl.eng.Engine] (executor-thread-1) Engine name: TensorFlow
2020-05-11 19:22:42,980 ERROR [io.qua.ver.htt.run.QuarkusErrorHandler] (executor-thread-1) HTTP Request to /predict failed, error id: 45121a3e-fbf8-4684-911c-4e9250ed8f41-1: java.lang.UnsatisfiedLinkError: org.tensorflow.internal.c_api.global.tensorflow.TF_Version()Lorg/bytedeco/javacpp/BytePointer; [symbol: Java_org_tensorflow_internal_c_1api_global_tensorflow_TF_1Version or Java_org_tensorflow_internal_c_1api_global_tensorflow_TF_1Version__]
        at com.oracle.svm.jni.access.JNINativeLinkage.getOrFindEntryPoint(JNINativeLinkage.java:145)
        at com.oracle.svm.jni.JNIGeneratedMethodSupport.nativeCallAddress(JNIGeneratedMethodSupport.java:57)
        at org.tensorflow.internal.c_api.global.tensorflow.TF_Version(tensorflow.java)
        at org.tensorflow.TensorFlow.version(TensorFlow.java:37)
        at ai.djl.tensorflow.engine.TfEngine.getVersion(TfEngine.java:64)
        at ai.djl.engine.Engine.debugEnvironment(Engine.java:171)
        at com.examples.ExampleService.<init>(ExampleService.java:42)

Expected Behavior

Running in GraalVM Native Image executable, the libaries should be loaded and usable through JNI bridge. I have proven this in the past with this PoC: https://github.com/murphye/quarkus-tensorflow-inception/blob/master/src/main/java/io/quarkus/tensorflow/LoadTensorFlow.java

Next Steps

I need some guidance on how TensorFlow is loaded in DJL if it’s not using System.loadLibrary as shown here: https://github.com/murphye/quarkus-tensorflow-inception/blob/master/src/main/java/io/quarkus/tensorflow/LoadTensorFlow.java#L98

How else does the TensorFlow library get loaded, and how can I further diagnose the issue when running in Native mode?

[murphye]

@stu1130 @frankfliu

Here is the format you need to follow:

--initialize-at-run-time=org.tensorflow.EagerSession\\,org.bytedeco.javacpp.Loader\\, (Add more as needed)

This is painful for the moment, but a better way can potentially be implemented for a Quarkus extension to handle this. For now, let’s go with --initialize-at-run-time

[stu1130]

@murphye Let us know if you encounter other issues when using --initialize-at-run-time and feel free to reopen it if you have any other question