question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

GPU - Out of Memory error - but GPU not fully utilised (1/2)

See original GitHub issue
  • face_recognition version: v1.2.2
  • Python version:3.6
  • Operating System:CentOS

Description

I am training a set of images on a GPU machine, but it fails with Out of Memory error - while checking on the GPU usage only one GPU is being used and it is full.

image

What I Did

Code:

face_bounding_boxes = face_recognition.face_locations(image, number_of_times_to_upsample=2, model="cnn")
X.append(face_recognition.face_encodings(image, known_face_locations=face_bounding_boxes, num_jitters=100)[0])

Images are less than 100kb and if I try to reduce upsample and num_jitters, it is impacting my accuracy. Why is it using only one GPU when I have 2 GPUs on my machine.

Please advice on this. Thank you.

Issue Analytics

  • State:open
  • Created 3 years ago
  • Comments:11 (2 by maintainers)

github_iconTop GitHub Comments

1reaction
essam-salahcommented, Apr 21, 2021

If you trying this repo, here are some details, hope it will be helpful:

  • PostgreSQL implementation of euclidean distance calculation works great - no need to train all the existing images for adding new registration - you can skip the entire training and classifier part - and PostgreSQL supports CUBE datatype.
  • Accuracy of Indian faces was average - I was able to improve this to some extent - by cropping the face and applying gamma scaling - this will remove unnecessary pixels from the image and improve the brightness - getting decent accuracy now.
  • if you are using PostgreSQL, you can reduce the error by comparing against specific user encodings, I have tried 1-N comparison on 25K stored encodings with good accuracy - however, for production deployments, I would prefer 1-1 comparison.

Thanks!

Hello rathishkumar,

I am trying to use this model to build an attendance system using face recognition but I have an issue with the accuracy. I am trying to reduce the distance threshold to ignore the false positive recognition. So can you please elaborate more how could you improve the accuracy of the model ?

0reactions
rathishkumarcommented, Apr 22, 2021

If you trying this repo, here are some details, hope it will be helpful:

  • PostgreSQL implementation of euclidean distance calculation works great - no need to train all the existing images for adding new registration - you can skip the entire training and classifier part - and PostgreSQL supports CUBE datatype.
  • Accuracy of Indian faces was average - I was able to improve this to some extent - by cropping the face and applying gamma scaling - this will remove unnecessary pixels from the image and improve the brightness - getting decent accuracy now.
  • if you are using PostgreSQL, you can reduce the error by comparing against specific user encodings, I have tried 1-N comparison on 25K stored encodings with good accuracy - however, for production deployments, I would prefer 1-1 comparison.

Thanks!

Hello rathishkumar,

I am trying to use this model to build an attendance system using face recognition but I have an issue with the accuracy. I am trying to reduce the distance threshold to ignore the false positive recognition. So can you please elaborate more how could you improve the accuracy of the model ?

I have shared my thoughts to a similar question here . Apart from these, I have used PostgreSQL for saving the encodings and computing the euclidean distance, I was trying to do a 1 - N ( N number of photos of a same person), so I was able to get good results. I have shared details here . Hope it helps.

Read more comments on GitHub >

github_iconTop Results From Across the Web

7 Tested Methods to Fix Your GPU Memory is Full Message
Your GPU memory is full warning appears when using your device to process and watch high-resolution videos. Don't hesitate to take action!
Read more >
Cuda out of memory but gpu memory is utilized about a half
The GPU doesn't flush the memory thinking the data is still usefull and this creates a problem when I do changes in the...
Read more >
How can I solve 'ran out of gpu memory' in TensorFlow
I was encountering out of memory errors when training a small CNN on a GTX 970. Through somewhat of a fluke, I discovered...
Read more >
Your GPU memory is full? Try these fixes to resolve it! - YouTube
Your GPU memory is full ? Try these fixes to resolve it!This video will show you how to do it!Try the following solutions...
Read more >
11 GB of GPU RAM used, and no process listed by nvidia-smi
It's probably the result of a corrupted context on the GPU, perhaps associated with your killed script. you can try using nvidia-smi to...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found