Slow progress using accelerate on multi-node

When using multi-GPU via the accelerate scripts, performance is improved, however when doing multi-node-multi-GPU performance degrades below usability.

Benchmarks:

1. Single P4 GPU: 1.8 it/sec Iteration: 100%|█████████████████████████████████████████████| 20/20 [00:26<00:00, 1.33s/it]
2. Dual P4 GPU (Same host): 2.21 it/sec Iteration: 70%|██████████████████████████████▍ | 14/20 [00:21<00:02, 2.21it/s]
3. Quad P4 GPU (Two hosts): 285.15 sec/it Iteration: 2%|██▊ | 12/500 [32:55<38:39:10, 285.15s/it]

Opening this issue as a way to track resolution for public information.

[deepglugs]

I was able to get reasonable results after switching my dataloader to use webdataset. There was likely a huge bottleneck in my custom dataloader that was causing 200+s/it even with only two GPUs. I was able to get 4 GPUs going, but noticed loss was 0.0 (and stays at 0.0). I’ll mess around with the learning rate and see if that helps (update: it didn’t).

[haukened]

Wait. What’s a vacation?

Sent from my iPhone

On Sep 8, 2022, at 6:20 PM, Phil Wang @.***> wrote:

@muellerzr have a great vacay! I need a vacation 🤔

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.