aspect ratio grouping error
See original GitHub issue❓ Questions and Help
I added a new loss and it works fine if I use a single GPU. However, it fails on “losses.backward()” if I use multiple GPUs. It seems this error relates to the “torch.distributed” The error information is below:
File "tools/train_net.py", line 170, in <module>
main()
File "tools/train_net.py", line 163, in main
model = train(cfg, args.local_rank, args.distributed)
File "tools/train_net.py", line 73, in train
arguments,
File "/home/maskrcnn_benchmark/engine/trainer.py", line 77, in do_train
losses.backward()
File "/usr/local/lib/python3.5/dist-packages/torch/tensor.py", line 102, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/usr/local/lib/python3.5/dist-packages/torch/autograd/__init__.py", line 90, in backward
allow_unreachable=True) # allow_unreachable flag
File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/deprecated/distributed.py", line 342, in reduction_fn_nccl
group=self.nccl_reduction_group_id)
File "/usr/local/lib/python3.5/dist-packages/torch/distributed/deprecated/__init__.py", line 317, in all_reduce_multigpu
return torch._C._dist_all_reduce_multigpu(tensor_list, op, group)
Issue Analytics
- State:
- Created 5 years ago
- Comments:21 (9 by maintainers)
Top Results From Across the Web
Post Error: ASPECT_RATIO_NOT_ALLOWED - WordStream HQ
If you've encountered an error message stating ASPECT_RATIO_NOT_ALLOWED, this error occurs when at least one of your logo images does not successfully crop...
Read more >aspect-ratio - CSS: Cascading Style Sheets - MDN Web Docs
The aspect-ratio CSS property sets a preferred aspect ratio for the box, which will be used in the calculation of auto sizes and...
Read more >Aspect ratio - CENOS Documentation
You can find this group named Aspect ratio error elements under Groups of Faces/Volumes. Aspect ratio error elements. Click the Eye icon (...
Read more >I have some problem with aspect ratio NaN - OutSystems
I'm having some problems with the aspect ratio in NaN, When I set the aspect ratio to NaN it gives me some error...
Read more >Powerpoint - re-scaling multiple images not working
1. I saw from the screenshot that the Lock Aspect Ratio checkbox is unchecked, so that's not the problem. 2. Try grouping the...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Oh, there might be indeed a problem with the
GroupedBatchSampler
. As a quick workaround, I’d recommend setting theASPECT_RATIO_GROUPING
to False in the config. I’ll need to dig a bit further to identify in which contexts the issue you are facing arises.That’s right. When setting the
ASPECT_RATIO_GROUPING
to False, everything is OK. I print the value ofmerged
in this line But I can not find any differences between using a single GPU and using multiple GPUs. multple GPUs:Single GPU: