[tfjs-react-native] Image classification of Livestream in real-time

System information

• TensorFlow.js version: 3.8.0
• tfjs-react-native version: 0.6.0
• Are you willing to contribute it (Yes/No): Yes

Describe the feature and the current behavior/state.

I wonder if there is a way to analyze the Livestream in real-time. For example, I’ve got a task to classify a mini object, such that the mobile’s camera is not capable of doing that. In this case, I have an IP camera, which could return a Livestream url for the app. However, currently, there is only the decodeJpeg method, which runs statically, not dynamically. Actually, I’m trying to snapshot on the Livestream frame and recursively calling decodeJpeg method, however, this is weird and I think the performance interacting with react-native is not good.

Will this change the current API? How?

Yes, and I think it needs two steps:

1. Allowing dynamic video streams to be the input (e.g. a Livestream url) of a model (which possibly will be a new API).
2. Allowing a real-time callback function for result display.

Who will benefit from this feature?

For users that need to use other cameras instead of mobile native cameras, because not all types of objects or images taken by phones are suitable for particular cases, thanks a lot!

[mattsoulanille]

Under the hood, tf.data.webcam still takes individual frames from the camera stream and transforms them into tensors. cameraWithTensors does the same thing.

I’m not sure how you’re displaying the livestream on the app, but if you’re rendering it in a WebGL2 context and know which texture corresponds to the video, you might be able to use fromTexture to get a tensor representing a single frame of the video. Then, you can run that through your model. The way cameraWithTensors works is it creates and expo-gl GLView and renders each frame to it. When the user requests the current frame as a tensor (or if it’s set to autoRender), it converts the WebGL texture into a tensor. You might be able to do something similar with your video feed.

Ideally, we would have a method similar to tf.browser.fromPixels that works on React Native components (probably implemented as a higher-order component like cameraWithTensors), but implementing that in React Native is likely a lot more complex than in the browser.

[google-ml-butler[bot]]

Closing as stale. Please @mention us if this needs more attention.