Why use torch.multiprocessing.spawn for distributed training

Hi there,

In the Swin UNETR scripts, e.g., https://github.com/Project-MONAI/research-contributions/blob/main/SwinUNETR/BRATS21/main.py, torch.multiprocessing.spawn is used for launching distributed training. Any reason why you didn’t use torch.distributed.launch? Did torch.multiprocessing.spawn give better performance than torch.distributed.launch for BraTS/BTCV-based Swin UNETR training?

Thanks!

[tangy5]

Thank you for clarification. Here are initial logs. single GPU, batch_size=1

2 GPUs, batch_size=2

multi GPU keeps taking longer time as number of GPUs increases. It will be worse if running with batch_size=1 on multi GPUs.

I mean, yes, when training with single GPU, the batch size is 1, then train on 2 GPUs, batch size is 2, the time is expected to be longer but should be less than 2 x time of Single GPU for each step/iteration. You could see 2 GPUs training is faster here, but is not exactly 2x faster, it’s ~1.7x faster.

[tangy5]

Hi @hw-ju , the SwinUNETR is tested of multi-GPU training with both DDP and MP Spawn. Both works well, no performance preference regarding different multi-GPU frameworks. You can safely use DDP. Thank you!