Update to the latest mediapipe facemesh model for tfjs?

System information

• TensorFlow.js version (you are using): 3.11.0
• Are you willing to contribute it (Yes/No): Yes

Describe the feature and the current behavior/state. The latest mediapipe facemesh model supports asymmetric facial expression and has a stunning performance on tracking the mouth area. Here’s a quick demo as of Jan 10 2022: https://codepen.io/mediapipe/full/KKgVaPJ (turn on the refine landmarks option). This is a huge improvement over the current pretrained model on tfjs, which can’t track the facial expression as lively: https://storage.googleapis.com/tfjs-models/demos/face-landmarks-detection/index.html?model=mediapipe_facemesh. I notice that tfjs landmark detection has been more than 1 year old on npm. Since tfjs landmark detection is using mediapipe facemesh underneath (constructor API attached below), would be great if we can update to the latest model:

const model = await faceLandmarksDetection.load(
    faceLandmarksDetection.SupportedPackages.mediapipeFacemesh)

Will this change the current api? How? No. It should be a swap of the underlying model to the latest one.

Who will benefit with this feature? Everyone who uses the facemesh module with tfjs. As shown in the link above, the model performance is visually much much better with the latest mediapipe facemesh release. Yet the inference time is almost identical.

[vladmandic]

update:

• new facemesh-detection-full has LOWER resolution (192 vs 256) vs old blazeface-back and generates far noisier output (2304 vs 896 candidates). neither i consider positive, so unless there is clear documentation why its better i’m skipping it.
• new facemesh-landmarks seems identical in every way to the old one except its quantized to f16 by default
• new facemesh-landmarks-attention does work without custom kernel ops and it does so almost as fast as standard landmarks model. but model output is totally undocumented (each attention sub-model produces separate output tensor which maps to what?), so will take some time to reverse-engineer it to something useful.

[vladmandic]

can you clarify?

last published facemesh model is quite old and hasn’t changed

• https://tfhub.dev/mediapipe/tfjs-model/facemesh/1/default/1

updated model is facemesh-with-attention, but afaik that is tflite only and requires mediapipe’s custom wasm backend for inference as model relies on several custom ops to achieve attention

• https://mediapipe.readthedocs.io/en/latest/solutions/face_mesh.html#attention-mesh-model

(specifically, ops are transform_landmarks, transform_tensor_bilinear, landmarks_to_transform_matrix and only implementation for them is C++ based one in mediapipe backend)

or did i miss something and there really is a new facemesh model that i cannot find?