Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Predict memory leak
See original GitHub issue🐛 Bug
The prediction contains a reference to the image array. If any stage of the workflow starts accumulating the predictions objects (as it currently happens in COCOMetric this will result in a memory leak.
COCOMetric currently uses a “quickfix” to make sure the images are not being stored (take a look at accumulate), but we need a more general solution.
Issue Analytics
- State:
- Created 3 years ago
- Comments:22 (12 by maintainers)
Top Results From Across the Web
memory leak in tf.keras.Model.predict #44711 - GitHub
So? I understand the OOM of GPU is because the GPU memory is too small. It means it is not the same issue...
Read more >tf.keras model.predict results in memory leak - Stack Overflow
I've found a fix for the memory leak. While K.clear_session() doesn't do anything in my case, adding a garbage collection after each call...
Read more >Huge memory leakage issue with tf.keras.models.predict()
After say 10000 such calls o predict(), while my MBP memory usage stays under 10GB, MACSTUDIO climbs to ~80GB (and counting up for...
Read more >Keras memory leak - The Kernel Trip
predict () . Another Github issue is simply called Memory leak . There even is another article simply titled Dealing with memory leak...
Read more >Dealing with memory leak issue in Keras model training
Going by the explanation from the solution above, another proposed solution I could find was replacing model.predict() with model.
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
a temoporary solution that seems to reduce memory leakage is to use num_workers not larger than the number of physical cores of the cpu as described here https://discuss.pytorch.org/t/num-workers-in-dataloader-will-increase-memory-usage/28522/9
might be linked with https://github.com/pytorch/pytorch/issues/13246 again, sorry Lucas 😄