tfjs-node works, tfjs-node-gpu fails with: Resource exhausted: OOM when allocating tensor with shape[2048]
See original GitHub issueTensorFlow.js version
tfjs-node-gpu: 0.1.18
Browser version
NodeJS v8.12.0, Windows 10, CUDA 9
Describe the problem or feature request
I have code in python which blurs an image, it works on Windows in Python on GPU.
I am trying to do the same from node, I have successfully converted the script totfjs-node
and it works and produces exactly the same results.
When I try to run tfjs-node-gpu
I go OOM, with the following error:
Resource exhausted: OOM when allocating tensor with shape[2048] and type float
log.txt
Command used to convert my model:
Miniconda3\Scripts\tensorflowjs_converter.exe --input_format=tf_frozen_model --output_node_names="image_tensor,detection_boxes,detection_scores,detection_classes,num_detections" --saved_model_tags=serve frozen_inference_graph.pb web_model
Weights: weights_manifest.json.txt
Sample code in python: sample.py.txt
Sample code in js: sample.js.txt
What I have tried:
- Tried with a very small input image, still failed.
Thoughts: Could this be a tensorflowjs_converter problem, where it expects the model to be ran in the browser, and converts to more generic types like int32 to store the data where the original code in python uses uint8.
This is how I pass the image to the model: tf.tensor4d(values, outShape, 'int32');
, as I understand there is no uint8 support ? also for some reason this operation takes ~40ms, is this expected ?
Issue Analytics
- State:
- Created 5 years ago
- Comments:15 (6 by maintainers)
Top GitHub Comments
I’m running into the same issue. I’m running a frozen model for inference on a video stream. It works for about 2 minutes then I get a crash. It seems always to be after a period of time regardless of the inputs. The inference graph processes around 15 images per seconds.
Here is a relevent part (I hope) of my error:
It works with tfjs-node instead of tfjs-node-gpu (0.1.19).
I’m using a GeForce GTX 1070
I no longer get this error using the latest version of @tensorflow/tfjs-node-gpu: 0.1.21