Question: What is the standard practice when performing inference on a video to maximise performance (FPS, Memory)?

TensorFlow.js version

Using this source for the script : https://cdn.jsdelivr.net/npm/@tensorflow/tfjs/dist/tf.min.js

Describe the question

I am trying to perform inference on a video and feed it back to a canvas. Current pipeline is:
VideoTag -> Hidden Canvas -> OpencvJS(matrix conversions and resizing) -> TFJS Predict -> Post Processing using OpencvJS -> Display Canvas

Major blockages in the pipeline are DrawImage function which draws image to the Hidden Canvas and post processing functions. Any suggestions on how we can improve this pipeline by using any TFJS libraries or any other libraries that are commonly used?

[annxingyuan]

Hi @sacredvoid - as Ping mentioned you should be able to achieve faster performance by directly rendering the TFJS output texture for post processing. The key is to initialize a custom WebGL backend with your own WebGL context, like so:

tf.registerBackend('webgl', () => {
    return new tf.webgl.MathBackendWebGL(
        new tf.webgl.GPGPUContext(canvasElement.getContext('webgl2', WEBGL_ATTRIBUTES)));
  });
await tf.setBackend('webgl');

This way, you can access the WebGL context that TFJS uses for computation and chain a rendering shader to the end of the inference pipeline. Within the rendering shader you could perform RGB -> YUV conversion, as described in this post: https://stackoverflow.com/questions/7901519/how-to-use-opengl-fragment-shader-to-convert-rgb-to-yuv420

[AbdallaGomaa]

Hi @sacredvoid. I am trying to develop a very similar application to the one you’re making. Did you get it working by setting up the same context for the canvas and TFjs? Any chance you have some sample code to help out? Would be much appreciated!!