Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Multiple GPU connected for training, but average utilization per GPU is lower than using single GPU
See original GitHub issueDescribe the bug Trained classification model with a single GPU, and saw average GPU utilization of ~40%.
I assumed using more GPU by increasing n_gpu in args can speed it up in proportional to the number of GPUs supplied.
In reality, the total GPU utilization remains the same at ~40%, but the average GPU utilization is ~7%.
Screenshots
Single GPU
Multiple GPUs
Desktop (please complete the following information):
-OS
4.15.0-124-generic #127-Ubuntu SMP Fri Nov 6 10:54:43 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Issue Analytics
- State:
- Created 3 years ago
- Comments:5 (2 by maintainers)
Top Results From Across the Web
Top Related Medium Post
Top Related StackOverflow Question
Troubleshoot Live Code
Top Related Reddit Thread
Top Related Hackernoon Post
Top Related Tweet
Top Related Dev.to Post
Top Related Hashnode Post
Sadly, this is probably because of issues with DataParallel from PyTorch. I am planning to upgrade to DistributedDataParallel, which should help. There’s no timeframe as of yet though.
Your GPU utilization should go up if you increase the batch size. Right now, your GPUs are getting data starved.