SoLoader can't find .so deps when app is packaged as Prebuilt APK with Android AOSP Source

Thanks so much for the hard work here!

I have a unique situation that perhaps you can shed some light on!

I am building a React Native Android application that will ship with an Android Device, and therefore it’s being included in the AOSP build process as a prebuilt APK.

When I install the APK via ADB, it runs great! However, when I build the Android source including the App, I flash the device, and see the App on the homescreen. However, when it boots, I see the following logs from SoLoader:

01-08 06:20:53.366 2950-2997/com.lightos D/SoLoader: Loading lib dependencies: [libglog_init.so, libyoga.so, libprivatedata.so, libfb.so, libfolly_json.so, libjsc.so, libglog.so, libgnustl_shared.so, libandroid.so, liblog.so, libstdc++.so, libm.so, libc.so, libdl.so]
    About to load: libglog_init.so
01-08 06:20:53.368 695-717/? D/ActivityTrigger: ActivityTrigger activityStopTrigger 
01-08 06:20:53.373 3430-3514/? W/Adreno-ES20: <core_glFinish:34>: glFinish skipped: 0
01-08 06:20:53.376 3430-3514/? W/Adreno-ES20: <core_glFinish:34>: glFinish skipped: 0
01-08 06:20:53.465 2950-2997/com.lightos D/SoLoader: Loading lib dependencies: [libglog.so, libgnustl_shared.so, liblog.so, libstdc++.so, libm.so, libc.so, libdl.so]
    About to load: libglog.so
01-08 06:20:53.515 2950-2997/com.lightos D/SoLoader: Loading lib dependencies: [libgnustl_shared.so, libstdc++.so, libm.so, libc.so, libdl.so]
01-08 06:20:53.516 2950-2997/com.lightos D/SoLoader: About to load: libgnustl_shared.so
01-08 06:20:53.550 2950-2997/com.lightos D/SoLoader: Loading lib dependencies: [libm.so, libc.so, libdl.so]
    About to load: libm.so
01-08 06:20:53.551 2950-2997/com.lightos D/SoLoader: libm.so not found on /data/data/com.lightos/lib-main
    Result 0 for libm.so in source com.facebook.soloader.ApkSoSource[root = /data/data/com.lightos/lib-main flags = 1]
    Extraction logs: null
01-08 06:20:53.552 2950-2997/com.lightos D/SoLoader: libm.so not found on /vendor/lib
    Result 0 for libm.so in source com.facebook.soloader.DirectorySoSource[root = /vendor/lib flags = 2]
01-08 06:20:53.553 2950-2997/com.lightos D/SoLoader: Loaded: libm.so
01-08 06:20:53.554 2950-2997/com.lightos D/SoLoader: About to load: libc.so
    libc.so not found on /data/data/com.lightos/lib-main
01-08 06:20:53.555 2950-2997/com.lightos D/SoLoader: Result 0 for libc.so in source com.facebook.soloader.ApkSoSource[root = /data/data/com.lightos/lib-main flags = 1]
01-08 06:20:53.556 2950-2997/com.lightos D/SoLoader: Extraction logs: null
    libc.so not found on /vendor/lib
01-08 06:20:53.557 2950-2997/com.lightos D/SoLoader: Result 0 for libc.so in source com.facebook.soloader.DirectorySoSource[root = /vendor/lib flags = 2]
    Loaded: libc.so
01-08 06:20:53.558 2950-2997/com.lightos D/SoLoader: About to load: libdl.so
    libdl.so not found on /data/data/com.lightos/lib-main
01-08 06:20:53.559 2950-2997/com.lightos D/SoLoader: Result 0 for libdl.so in source com.facebook.soloader.ApkSoSource[root = /data/data/com.lightos/lib-main flags = 1]
    Extraction logs: null
01-08 06:20:53.560 2950-2997/com.lightos D/SoLoader: libdl.so not found on /vendor/lib
    Result 0 for libdl.so in source com.facebook.soloader.DirectorySoSource[root = /vendor/lib flags = 2]
    Loaded: libdl.so
01-08 06:20:53.563 2950-2997/com.lightos E/SoLoader: Error when loading lib: dlopen failed: couldn't map "/data/data/com.lightos/lib-main/libgnustl_shared.so" segment 2: Permission denied
01-08 06:20:53.564 2950-2997/com.lightos E/SoLoader: Could not load: libgnustl_shared.so
    Could not load: libglog.so
    Could not load: libglog_init.so
01-08 06:20:53.565 2950-2997/com.lightos E/SoLoader: Could not load: libreactnativejni.so
01-08 06:20:53.570 2950-2997/com.lightos E/AndroidRuntime: FATAL EXCEPTION: Thread-2
    Process: com.lightos, PID: 2950
    java.lang.UnsatisfiedLinkError: couldn't find DSO to load: libreactnativejni.so
        at com.facebook.soloader.SoLoader.doLoadLibraryBySoName(SoLoader.java:522)
        at com.facebook.soloader.SoLoader.loadLibraryBySoName(SoLoader.java:420)
        at com.facebook.soloader.SoLoader.loadLibrary(SoLoader.java:370)
        at com.facebook.soloader.SoLoader.loadLibrary(SoLoader.java:335)
        at com.facebook.react.bridge.ReactBridge.staticInit(ReactBridge.java:18)
        at com.facebook.react.bridge.NativeMap.<clinit>(NativeMap.java:19)
        at com.facebook.react.bridge.JSCJavaScriptExecutorFactory.create(JSCJavaScriptExecutorFactory.java:21)
        at com.facebook.react.ReactInstanceManager$5.run(ReactInstanceManager.java:917)
        at java.lang.Thread.run(Thread.java:764)
01-08 06:20:53.547 2950-2950/com.lightos W/Thread-2: type=1400 audit(0.0:74): avc: denied { execute } for path="/data/data/com.lightos/lib-main/libgnustl_shared.so" dev="mmcblk0p38" ino=2430 scontext=u:r:system_app:s0 tcontext=u:object_r:system_app_data_file:s0 tclass=file permissive=0

It appears to me that the install process that moves the .so files to the appropriate place on the Android filesystem isn’t being run for the packaged app in the AOSP build step (but I have confirmed they are in the APK itself by unzipping and checking).

What should I do to get this working as an Android prebuilt?

Thank you!

[hhff]

Cool yeah. I have a feeling that “permission denied” is actually because the file doesn’t exist. I’m still working on this, but when I solve it I’ll come back and update this issue to help anyone else who comes this way.

[hhff]

hi @SergiiGudym - yup I did. So - for anyone else:

If you’re embedding your APK into a AOSP build that will packaged with the ROM, it’s important to note a few things:

• the APK isn’t going to go through an “Install” process (like it would via ADB or any other traditional install process). As part of that normal install process, packaged, prebuilt libs will be extracted and moved to the appropriate place for the APK to require them. This doesn’t happen when packaging the APK with AOSP.

• an APK is basically a fancy zip file, and those static libs are inside.

---

So - when you’re packaging your React Native app with an AOSP ROM, you’ll need to do the following:

1. Do all of the normal setup you would with a standard APK that you’re building with AOSP (some googling will help with this)

2. Unzip your APK:

mkdir ./unzipped && unzip ./MyAPK.apk -d ./unzipped

1. Move your static libs to the same level as your Android.mk (assuming you’re building for both architectures as per react native default)

mv ./unzipped/lib/armeabi-v7a ./lib
mv ./unzipped/lib/x86 ./lib64

1. Edit your Android.mk file to move these libs during AOSP build time:

LOCAL_PATH := $(call my-dir)

# Example for 32 bit deps
include $(CLEAR_VARS)
LOCAL_MODULE := libfb
LOCAL_SRC_FILES := lib/libfb.so
LOCAL_MODULE_TAGS := optional
LOCAL_MULTILIB := 32
LOCAL_MODULE_SUFFIX := .so
LOCAL_MODULE_CLASS := SHARED_LIBRARIES
LOCAL_MODULE_PATH := $(PRODUCT_OUT)/vendor/lib
include $(BUILD_PREBUILT)

# Example for 64 bit deps
include $(CLEAR_VARS)
LOCAL_MODULE := libfb
LOCAL_SRC_FILES := lib64/libfb.so
LOCAL_MODULE_TAGS := optional
LOCAL_MULTILIB := 64
LOCAL_MODULE_SUFFIX := .so
LOCAL_MODULE_CLASS := SHARED_LIBRARIES
LOCAL_MODULE_PATH := $(PRODUCT_OUT)/vendor/lib64
include $(BUILD_PREBUILT)

# ^^At the time of writing, React Native has like ~11 static libs, so you'll have ~22 of these blocks

# Finally, declare the APK itself
include $(CLEAR_VARS)
LOCAL_MODULE_TAGS := optional
LOCAL_MODULE := MyAPK
LOCAL_SRC_FILES := $(LOCAL_MODULE).apk
LOCAL_MODULE_CLASS := APPS
LOCAL_MODULE_SUFFIX := $(COMMON_ANDROID_PACKAGE_SUFFIX)
LOCAL_CERTIFICATE := platform
LOCAL_PRIVILEGED_MODULE := true
include $(BUILD_PREBUILT)

1. Now when you build AOSP, flash the device, and run MyAPK, SoLoader should be able to find and link these libs at runtime.

---

Some gotchas:

• The Android.mk file is super finicky. Make sure you don’t have any extra spaces at the end of lines, etc. Seriously! It can crash an AOSP build.
• Remember your static libs may change if your build does (Upgrading react native, etc). We’ve setup a shell script to pull in the latest APK from an s3 bucket, and automate most of the above. Then when we’re about to check in the new version of the APK, we’ll see any libs were added. If so, we’ll add a block for them in the Android.mk before committing.
• Hit me with any more questions!