'MediaPipeException: MediaPipe Aborted' on Android

Plugin Version or Commit ID

v0.10.1

Unity Version

2021.3.10f1

Your Host OS

Windows 10

Target Platform

Android

[Windows Only] Visual Studio C++ and Windows SDK Version

No response

[Linux Only] GCC/G++ and GLIBC Version

No response

[Android Only] Android Build Tools and NDK Version

Build Tools: 30.0.3 NDK. r21d

[iOS Only] XCode Version

No response

Command Sequences

HolisticGPU.txt holistic_graph_gpu.txt

Log

MediaPipeException: MediaPipe Aborted, refer glog files for more details at Mediapipe.MpReturnCodeExtension.Assert (Mediapipe.MpReturnCode code) [0x0002f] in <d24e9f466dc24aec8932e0c7beddfafc>:0 at Mediapipe.CalculatorGraph…ctor (System.Byte[] serializedConfig) [0x00012] in <d24e9f466dc24aec8932e0c7beddfafc>:0 at Mediapipe.CalculatorGraph…ctor (Mediapipe.CalculatorGraphConfig config) [0x00007] in <d24e9f466dc24aec8932e0c7beddfafc>:0 at Mediapipe.CalculatorGraph…ctor (System.String textFormatConfig) [0x0000c] in <d24e9f466dc24aec8932e0c7beddfafc>:0 at Mediapipe.Unity.Test.HolisticGPU+<Start>d__12.MoveNext () [0x001a9] in <404d0fb366b74c33b334f1a2136a623b>:0 at UnityEngine.SetupCoroutine.InvokeMoveNext (System.Collections.IEnumerator enumerator, System.IntPtr returnValueAddress) [0x00026] in <a960d8def81c47b88e2442adc0b94113>:0

full log.txt

Additional Context

I want to run MediaPipe Holistic on Android. The goal of the project is to draw body, face and hand landmarks on frames and get landmark coordinates. So, I have built an android apk file in unity. But when I run the app on android smartphone it shows the blank screen. In the android logcat I see the following error: MediaPipeException: MediaPipe Aborted, refer glog files for more details

In unity I used MediaPipeUnityPlugin pre-built package released here.

At first I have created c# script and the graph to run MediaPipe Holistic on CPU. Hovewer, I later revealed this issue and thought it might be mandatory to use GPU on android. So, I changed the files, but got the same result.

The Unity scene contains a canvas screen and an empty gameobject. I attached c# script named ‘HolisticGPU’ and a graph named ‘holistic_graph_gpu’ to that gameobject. These files are attached to the issue.

I wonder what might be the cause of this error. Thank you in advance!

[G93-7]

I have validated the graph config as advised by @homuler.

//_graph = new CalculatorGraph(_configAsset.text);

var config = CalculatorGraphConfig.Parser.ParseFromTextFormat(_configAsset.text);
Debug.Log("before validatedGraphConfig");
using (var validatedGraphConfig = new ValidatedGraphConfig())
{
    Debug.Log("before Initialize");
    validatedGraphConfig.Initialize(config).AssertOk();
    Debug.Log("before CalculatorGraph");
    _graph = new CalculatorGraph(validatedGraphConfig.Config());
}

After that, I got the following error in a bit more understandable form:

MediaPipeException: UNKNOWN: Input stream "transformed_input_video_gpu" of calculator "[ImagePropertiesCalculator, input stream: IMAGE:transformed_input_video_gpu, and output stream: SIZE:image_size]" expects packets of type "OneOf<::mediapipe::Image, mediapipe::ImageFrame>" but the connected output stream will contain packets of type "mediapipe::GpuBuffer"

I have found that a calculator named ImagePropertiesCalculator expects packets of type Image or ImageFrame but I have passed a packet of type GpuBuffer. So, I have made transformed_input_video the input of ImagePropertiesCalculator instead of transformed_input_video_gpu. After running the graph I found that the same problem was true for AnnotationOverlayCalculator. So, I have changed the sequence of nodes. I put GpuBufferToImageFrameCalculator after HolisticLandmarkGpu calculator. It returned transformed_input_video_imageframe which is of type Imageframe. I have attached it to AnnotationOverlayCalculator.

After that I got the following errors:

MediaPipeException: FAILED_PRECONDITION: Graph has errors: 
Calculator::Open() for node "holisticlandmarkgpu__poselandmarkgpu__posedetectiongpu__inferencecalculator__holisticlandmarkgpu__poselandmarkgpu__posedetectiongpu__InferenceCalculator" failed: Failed to read /storage/emulated/0/Android/data/com.DefaultCompany.TestHolisticPCGPU/files/pose_detection.bytes
Calculator::Open() for node "holisticlandmarkgpu__handlandmarksleftandrightgpu__handlandmarksfromposegpu_1__handlandmarkgpu__handlandmarkmodelloader__LocalFileContentsCalculator" failed: Failed to read /storage/emulated/0/Android/data/com.DefaultCompany.TestHolisticPCGPU/files/hand_landmark_full.bytes
Calculator::Open() for node "holisticlandmarkgpu__poselandmarkgpu__poselandmarkbyroigpu__poselandmarkmodelloader__LocalFileContentsCalculator" failed: Failed to read /storage/emulated/0/Android/data/com.DefaultCompany.TestHolisticPCGPU/files/pose_landmark_lite.bytes
Calculator::Open() for node "holisticlandmarkgpu__facelandmarksfromposegpu__facelandmarkgp

I guessed that it could not load model files. So, I have loaded them manually in the script.

_resourceManager = new StreamingAssetsResourceManager();
yield return _resourceManager.PrepareAssetAsync("pose_detection.bytes", "pose_detection.bytes", false);
yield return _resourceManager.PrepareAssetAsync("pose_landmark_lite.bytes", "pose_landmark_lite.bytes", false);
yield return _resourceManager.PrepareAssetAsync("hand_landmark_full.bytes", "hand_landmark_full.bytes", false);
yield return _resourceManager.PrepareAssetAsync("hand_recrop.bytes", "hand_recrop.bytes", false);
yield return _resourceManager.PrepareAssetAsync("handedness.txt", "handedness.txt", false);
yield return _resourceManager.PrepareAssetAsync("face_landmark.bytes", "face_landmark.bytes", false);
yield return _resourceManager.PrepareAssetAsync("face_detection_short_range.bytes", "face_detection_short_range.bytes", false);
yield return _resourceManager.PrepareAssetAsync("face_landmark_with_attention.bytes", "face_landmark_with_attention.bytes", false);

And finally it worked!

The edited script and graph config files are attached to the comment. HolisticGPU.txt holistic_graph_gpu_edited.txt

Thanks @homuler for the comment. It helped me find the reason of the error.

[homuler]

Yes, of course, I have tried the sample app. It works fine in the editor.

I mean, did you try the sample app on Android? Is the screen image rotated there, too?

The _screen gameobject displays an image with resolution _width x _height (640X480 in this case). As I guessed, AutoFit.cs script enlarges the image to fit the width or height of the screen. However, the quality of the image deteriorates. Is it possible to display the original image returned from the camera (with an unchanged resolution) and then draw landmarks on it?

If you think there is a problem with AutoFit.cs, please file another issue. However, it seems to me that there’s a contradiction between what you are saying and what you are trying to do.

If you want to display the original image, then you must not resize the image. In this case, when the image size is small, it is expected that it will not fill the screen. On the other hand, if you want a small image to fill the screen, you need to resize it. However, if the input image (e.g. 640x480) is smaller than the screen size (e.g. 2400x1080), the resized image should be rough, of course. If you are still unhappy with the coarseness of the resized image, you need to increase the resolution of the camera.

And I think those things are not related to MediaPipe at all.