Save tensors from react-native cameraWithTensors api to jpeg

Please make sure that this is a feature request. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:feature_template

System information

• TensorFlow.js version (you are using):

  “@tensorflow/tfjs”: “2.0.0”, “@tensorflow/tfjs-backend-webgl”: “2.0.0”, “@tensorflow/tfjs-core”: “2.0.0”, “@tensorflow/tfjs-react-native”: “0.3.0”,

• Are you willing to contribute it (Yes/No): Yes

Describe the feature and the current behavior/state.

Currently when using the cameraWithTensor HOC one can handle directly the underlying tensor to make e.g. real-time prediction. It would be nice to be able to store the stream of tensors as jpeg or video for instance for later use or visualization.

I have created a repo to showcase my currently unsuccessful attempts: https://github.com/ClementWalter/tfjs-expo-example

Will this change the current api? How?

Who will benefit with this feature?

Any Other info.

[ClementWalter]

yes! actually if you go to https://github.com/ClementWalter/tfjs-expo-example/blob/master/utils.js the function does the job. I will try to push it as a contrib to tfjs-react-native very soon. However, as I have said above, I don’t really understand the magic numbers used for width and height in the cameraWithTensor module.

[0xori]

I’m assuming it depends on the device. you can actually get the device available picture sizes. take a look at expo-camera docs. when I’m changing the dimensions according to the results, I’m getting better capture proportions.

 <TensorCamera
                        // Standard Camera props
                        ref={(ref) => this.camera = ref}
                        style={{ position: 'absolute',
                            left: 0,
                            top: 0,
                            width: 152,
                            height: 200,
                            zIndex: 0,}}
                        type={Camera.Constants.Type.front}
                        // Tensor related props
                        cameraTextureHeight={textureDims.height}
                        cameraTextureWidth={textureDims.width}
                        resizeHeight={200}
                        resizeWidth={152}
                        resizeDepth={3}
                        onReady={this.handleCameraStream}
                        autorender={true}
                    />

 handleCameraStream = async (images, updatePreview, gl)  =>{
      if(this.camera){
          console.log("camera ref")
          const ratios = await this.camera.camera.getSupportedRatiosAsync()
          console.log("ratios", ratios)
          const sizes = await this.camera.camera.getAvailablePictureSizesAsync("4:3");
          console.log("sizes", sizes)
      }

    const loop = async () => {
      let next = images.next();
      const nextImageTensor = next.value

      //
      // do something with tensor here
      //

      // if autorender is false you need the following two lines.
      // updatePreview();
      // gl.endFrameEXP();

    }
    loop();
  }