GPU - Out of Memory error - but GPU not fully utilised (1/2)
See original GitHub issue- face_recognition version: v1.2.2
- Python version:3.6
- Operating System:CentOS
Description
I am training a set of images on a GPU machine, but it fails with Out of Memory error - while checking on the GPU usage only one GPU is being used and it is full.
What I Did
Code:
face_bounding_boxes = face_recognition.face_locations(image, number_of_times_to_upsample=2, model="cnn")
X.append(face_recognition.face_encodings(image, known_face_locations=face_bounding_boxes, num_jitters=100)[0])
Images are less than 100kb and if I try to reduce upsample and num_jitters, it is impacting my accuracy. Why is it using only one GPU when I have 2 GPUs on my machine.
Please advice on this. Thank you.
Issue Analytics
- State:
- Created 3 years ago
- Comments:11 (2 by maintainers)
Top Results From Across the Web
7 Tested Methods to Fix Your GPU Memory is Full Message
Your GPU memory is full warning appears when using your device to process and watch high-resolution videos. Don't hesitate to take action!
Read more >Cuda out of memory but gpu memory is utilized about a half
The GPU doesn't flush the memory thinking the data is still usefull and this creates a problem when I do changes in the...
Read more >How can I solve 'ran out of gpu memory' in TensorFlow
I was encountering out of memory errors when training a small CNN on a GTX 970. Through somewhat of a fluke, I discovered...
Read more >Your GPU memory is full? Try these fixes to resolve it! - YouTube
Your GPU memory is full ? Try these fixes to resolve it!This video will show you how to do it!Try the following solutions...
Read more >11 GB of GPU RAM used, and no process listed by nvidia-smi
It's probably the result of a corrupted context on the GPU, perhaps associated with your killed script. you can try using nvidia-smi to...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Hello rathishkumar,
I am trying to use this model to build an attendance system using face recognition but I have an issue with the accuracy. I am trying to reduce the distance threshold to ignore the false positive recognition. So can you please elaborate more how could you improve the accuracy of the model ?
I have shared my thoughts to a similar question here . Apart from these, I have used PostgreSQL for saving the encodings and computing the euclidean distance, I was trying to do a 1 - N ( N number of photos of a same person), so I was able to get good results. I have shared details here . Hope it helps.