Help with Pose Detection tutorial setup

Plugin Version or Commit ID

latest

Unity Version

2021.2.7f1

Your Host OS

mac OS Monterey

Target Platform

UnityEditor

Description

Hello I’m trying to create separate model for pose detection by tweaking steps from face mesh tutorial. I’ve used this config https://github.com/google/mediapipe/blob/c6c80c37452d0938b1577bd1ad44ad096ca918e0/mediapipe/graphs/pose_tracking/pose_tracking_cpu.pbtxt

I’ve copy pasted code from FaceMesh.cs and changed to resourcemanager to pose_detection.bytes yield return _resourceManager.PrepareAssetAsync("pose_detection.bytes"); Code is attached PoseEstimation.cs.txt

however following error comes up MediaPipeException: INTERNAL: Graph has errors: Calculator::Process() for node "poserenderercpu__RecolorCalculator" failed: ; RET_CHECK failure (external/com_google_mediapipe/mediapipe/calculators/image/recolor_calculator.cc:241) input_mat.channels() == 3

Also attached files. I guess it has something to do with image texture channels or something any ideas how to solve this?

Code to Reproduce the issue

pose_tracking_cpu.txt PoseEstimation.cs.txt

Additional Context

No response

[homuler]

I’ve use this code however it throws some exception. at /NormalizedLandmarkListVectorPacket.cs:29

Which kind of exception occured? Please share the complete error message.

var poseLandmarksStream = new OutputStream<NormalizedLandmarkListVectorPacket, List<NormalizedLandmarkList>>(_graph, "pose_landmarks");

I think the stream type is wrong.

var poseLandmarksStream = new OutputStream<NormalizedLandmarkListPacket, NormalizedLandmarkList>(_graph, "pose_landmarks");

Also will appreciate if you can share documentation links maybe.

What document are you talking about?

[homuler]

The most straightforward way is to change the input format from RGBA to RGB.

@@ -43,9 +43,9 @@ namespace Mediapipe.Unity.Tutorial
 
       _screen.rectTransform.sizeDelta = new Vector2(_width, _height);
 
-      _inputTexture = new Texture2D(_width, _height, TextureFormat.RGBA32, false);
+      _inputTexture = new Texture2D(_width, _height, TextureFormat.RGB24, false);
       _inputPixelData = new Color32[_width * _height];
-      _outputTexture = new Texture2D(_width, _height, TextureFormat.RGBA32, false);
+      _outputTexture = new Texture2D(_width, _height, TextureFormat.RGB24, false);
       _outputPixelData = new Color32[_width * _height];
 
       _screen.texture = _outputTexture;
@@ -67,7 +67,7 @@ namespace Mediapipe.Unity.Tutorial
       while (true)
       {
         _inputTexture.SetPixels32(_webCamTexture.GetPixels32(_inputPixelData));
-        var imageFrame = new ImageFrame(ImageFormat.Types.Format.Srgba, _width, _height, _width * 4, _inputTexture.GetRawTextureData<byte>());
+        var imageFrame = new ImageFrame(ImageFormat.Types.Format.Srgb, _width, _height, _width * 3, _inputTexture.GetRawTextureData<byte>());
         var currentTimestamp = stopwatch.ElapsedTicks / (System.TimeSpan.TicksPerMillisecond / 1000);
         _graph.AddPacketToInputStream("input_video", new ImageFramePacket(imageFrame, new Timestamp(currentTimestamp))).AssertOk();

Another option is to stop rendering annotations using MediaPipe (see the Pose Tracking sample).

Also will appreciate if you can share of clarify whenever code for getting landmark coordinates is the same as in tutorial.

Sorry, I don’t get what you mean by “whenever code”.