BodyPix tflite model support

@Volcomix , your work is really impressive! Thank you so much for this work.

I saw this comment from @PINTO0309 in this issue https://github.com/Volcomix/virtual-background/issues/2#issuecomment-829730723

I don’t know if it will be useful for you, but I have converted and quantized it for various frameworks and committed it to my repository.

TFLite Float32/Float16/INT8, TFJS, TF-TRT, ONNX, CoreML, OpenVINO IR FP32/FP16, Myriad Inference Blob

https://github.com/PINTO0309/PINTO_model_zoo https://github.com/PINTO0309/PINTO_model_zoo/tree/main/109_Selfie_Segmentation

And I’m able to easily play with the different models. Thank you both for your hardwork.

I noticed @PINTO0309 also has bodypix tflite models here but they don’t seem to work with @Volcomix pipeline. Getting some logs I noticed the output image has the following:

console.log({
  inputHeight: this._tflite._getInputHeight(),
  inputWidth: this._tflite._getInputWidth(),
  inputChannelCount: this._tflite._getInputChannelCount(),
  outputHeight: this._tflite._getOutputHeight(),
  outputWidth: this._tflite._getOutputWidth(),
  outputChannelCount: this._tflite._getOutputChannelCount(),
});
// Outputs an image with 10x8 resolution, 17 channels

As you can see, the output seems wrong. Any advice on what to adjust? My intention is I want to run bodypix model in wasm hoping to gain more performance than the tfjs one. Is this something you can help with? Thank you in advance!

[PINTO0309]

All the models were re-transformed and totally replaced with the same model structure of TensorFlow.js. https://github.com/PINTO0309/PINTO_model_zoo/tree/main/035_BodyPix

[Volcomix]

After loading the new tflite models and trying with the fastest one (mobilenet050, stride16, 240x320, float16 quantization), the performance seems equivalent/worse with tflite on wasm than with tfjs on webgl. Moreover the tfjs one works on a higher resolution:

After checking other projects code and the tfjs implementation of BodyPix more closely, I would have to handle padding and scaling implied by the strides which is implemented using tensorflow in all the reference projects but would require a lot of extra work in this demo. As the performance results are not that good with the tflite model of BodyPix, I don’t wish to invest more time on this specific experiment.

I’m very sorry PINTO for making you spend time on it without ending.

P.S.: There is a work in progress in TF.js to handle tflite models. Maybe is there a chance that BodyPix would work on it when it will be ready: https://github.com/tensorflow/tfjs/tree/master/tfjs-tflite