Can TensorFlow React Native works with local models ?

I meet some problems with tfjs-react-native (https://github.com/tensorflow/tfjs/tree/master/tfjs-react-native).

Is the app created using expo? If so is it a managed or bare app? Yes, bare app.

Which version of react native and the dependencies in the install instructions above are you using? dependencies.txt

What device(s) are you running on? Note that not all simulators support webgl and thus may not work with tfjs-react-native. On my ASUS ROG Phone 2

What error messages are you seeing? Are there any relevant messages in the device logs? My app crash, but see the text below for more informations.

How could this bug be reproduced? Is there an example repo we can use to replicate the issue? I don’t have an example repo, please see my explanation below.

First of all, I followed the tutorials https://heartbeat.fritz.ai/image-classification-on-react-native-with-tensorflow-js-and-mobilenet-48a39185717c and https://medium.com/swlh/object-detection-with-android-phone-using-tensor-flow-lite-9463a1522ffd and it works perfectly fine on my app. The only main diff is the source of images. I use expo-camera takePictureAsync function : https://docs.expo.io/versions/latest/sdk/camera/#takepictureasync.

However, I looked for a functional example of tfjs-react-native use with a local model, and so a call of https://js.tensorflow.org/api_react_native/latest/#bundleResourceIO like in https://blog.tensorflow.org/2020/02/tensorflowjs-for-react-native-is-here.html (with burger_not_burger.json and burger_not_burger_weights.bin) but I didn’t find anything (except theses burgers but without the model files so… I can’t test it). I tried to do the same with https://tfhub.dev/tensorflow/tfjs-model/ssd_mobilenet_v2/1/default/1 :

• Downloaded the file and extract it
• Got the js_model/model.json file
• Used the command “cat” to concatenate group1-shard1of17 to group1-shard17of17 into a 65 765ko model_weights.bin
• Tried to load the model by using tf.loadGraphModel(bundleResourceIO(modelJson, modelWeightsId)) or tf.loadLayersModel(bundleResourceIO(modelJson, modelWeightsId)) but my app crash trying to load it and without showing anything into the catch block.

Is the file too big for react-native ? If true how you can handle a local model ? The model.json / model_weights.bin is always big and we can’t use any other format such as SavedModel, frozenModel or tflite. I false do you any idea what could go wrong here ?

On the other hand I tried to use a local classification model too : https://tfhub.dev/google/tfjs-model/imagenet/mobilenet_v2_035_224/classification/3/default/1 I did the same :

• Downloaded the file and extract it
• Got the model.json file
• Used the command “cat” to concatenate group1-shard1of2 to group1-shard2of2 into a 6 529ko model_weights.bin And into my app :
• Loaded my model with tf.loadGraphModel(bundleResourceIO(modelJson, modelWeightsId)) It works
• Trying to classify the image returned by expo-camera with this code : const imageAssetPath = Image.resolveAssetSource({ uri }) const imgB64 = await FileSystem.readAsStringAsync(imageAssetPath.uri, { encoding: FileSystem.EncodingType.Base64, }) const imgBuffer = tf.util.encodeString(imgB64, 'base64').buffer; const raw = new Uint8Array(imgBuffer) const imageTensor = decodeJpeg(raw) const predictions = model.classify(imageTensor) And got the following error : Error: The shape of dict[‘images’] provided in model.execute(dict) must be [-1,224,224,3], but was [300,300,3] It seems pretty classic, I’ve seen this error on numerous posts and I did try a lot of things : to reshape or resize my tensor but I never successed to make it work… Could you tell me how to make it work ?

Am I doing something wrong ? I start to think only pre-trained models (https://github.com/tensorflow/tfjs-models ) work with tfjs react-native ^^’ That’s pretty sad because I wanted to train my own models and use it into my app.

I also saw this post but there was not really an answer : https://github.com/tensorflow/tfjs/issues/3070

Thank you in advance for your answer, and have a good day.

[tafsiri]

@joaorenatojr87 that does look like it may be an issue unrelated to the sharding. to make this easier for us to track/resolve, could you make a new issue with everything your last comment as well as your model.json file.

[google-ml-butler[bot]]

Closing as stale. Please @mention us if this needs more attention.