Single machine with multiple GPU raise Found at least two devices Error

I am running on a single machine with multiple GPU

and raise Error
-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/home/w29758143/anaconda3/envs/pycorrector/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
  File "/home/w29758143/anaconda3/envs/pycorrector/lib/python3.7/site-packages/fairseq_cli/train.py", line 300, in distributed_main
    main(args, init_distributed=True)
  File "/home/w29758143/anaconda3/envs/pycorrector/lib/python3.7/site-packages/fairseq_cli/train.py", line 87, in main
    train(args, trainer, task, epoch_itr)
  File "/home/w29758143/anaconda3/envs/pycorrector/lib/python3.7/site-packages/fairseq_cli/train.py", line 130, in train
    log_output = trainer.train_step(samples)
  File "/home/w29758143/anaconda3/envs/pycorrector/lib/python3.7/site-packages/fairseq/trainer.py", line 400, in train_step
    assert all(norm == prev_norms[0] for norm in prev_norms) or all(
  File "/home/w29758143/anaconda3/envs/pycorrector/lib/python3.7/site-packages/fairseq/trainer.py", line 400, in <genexpr>
    assert all(norm == prev_norms[0] for norm in prev_norms) or all(
  File "/home/w29758143/anaconda3/envs/pycorrector/lib/python3.7/site-packages/torch/tensor.py", line 27, in wrapped
    return f(*args, **kwargs)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!

Below is my shell script

> GPU_ID=0,1,2,3
> DATA_BIN_DIR="output/bin"
> OUT_DIR="output/models"
> BATCH_SIZE=128
> MAX_TOKENS=10000
> SEED=1
> 
> CUDA_VISIBLE_DEVICES="${GPU_ID}" python3 -m fairseq_cli.train \
>     ${DATA_BIN_DIR} \
>     --save-dir ${OUT_DIR} \
>     -a fconv \
>     --num-workers=4 --skip-invalid-size-inputs-valid-test \
>     --encoder-embed-dim 300 \
>     --decoder-embed-dim 300 \
>     --decoder-out-embed-dim 500 \
>     --encoder-layers '[(1024,3)] * 5' --decoder-layers '[(1024,3)] * 5' \
>     --dropout='0.2' --clip-norm=0.1 \
>     --optimizer nag --momentum 0.99 \
>     --lr-scheduler=reduce_lr_on_plateau --lr=0.25 --lr-shrink=0.1 --min-lr=1e-4 \
>     --max-epoch 100 \
>     --batch-size ${BATCH_SIZE} \
>     --max-tokens ${MAX_TOKENS} \
>     --seed ${SEED}
>     

Am I missing anything?

python version 3.7.6 fairseq version 0.9.0 CUDA Version: 11.0
pyTorch VERSION: 1.7.1

[Hexa4C]

I am running on a single machine with multiple GPU

and raise Error
-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/home/w29758143/anaconda3/envs/pycorrector/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
  File "/home/w29758143/anaconda3/envs/pycorrector/lib/python3.7/site-packages/fairseq_cli/train.py", line 300, in distributed_main
    main(args, init_distributed=True)
  File "/home/w29758143/anaconda3/envs/pycorrector/lib/python3.7/site-packages/fairseq_cli/train.py", line 87, in main
    train(args, trainer, task, epoch_itr)
  File "/home/w29758143/anaconda3/envs/pycorrector/lib/python3.7/site-packages/fairseq_cli/train.py", line 130, in train
    log_output = trainer.train_step(samples)
  File "/home/w29758143/anaconda3/envs/pycorrector/lib/python3.7/site-packages/fairseq/trainer.py", line 400, in train_step
    assert all(norm == prev_norms[0] for norm in prev_norms) or all(
  File "/home/w29758143/anaconda3/envs/pycorrector/lib/python3.7/site-packages/fairseq/trainer.py", line 400, in <genexpr>
    assert all(norm == prev_norms[0] for norm in prev_norms) or all(
  File "/home/w29758143/anaconda3/envs/pycorrector/lib/python3.7/site-packages/torch/tensor.py", line 27, in wrapped
    return f(*args, **kwargs)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!

Below is my shell script

> GPU_ID=0,1,2,3
> DATA_BIN_DIR="output/bin"
> OUT_DIR="output/models"
> BATCH_SIZE=128
> MAX_TOKENS=10000
> SEED=1
> 
> CUDA_VISIBLE_DEVICES="${GPU_ID}" python3 -m fairseq_cli.train \
>     ${DATA_BIN_DIR} \
>     --save-dir ${OUT_DIR} \
>     -a fconv \
>     --num-workers=4 --skip-invalid-size-inputs-valid-test \
>     --encoder-embed-dim 300 \
>     --decoder-embed-dim 300 \
>     --decoder-out-embed-dim 500 \
>     --encoder-layers '[(1024,3)] * 5' --decoder-layers '[(1024,3)] * 5' \
>     --dropout='0.2' --clip-norm=0.1 \
>     --optimizer nag --momentum 0.99 \
>     --lr-scheduler=reduce_lr_on_plateau --lr=0.25 --lr-shrink=0.1 --min-lr=1e-4 \
>     --max-epoch 100 \
>     --batch-size ${BATCH_SIZE} \
>     --max-tokens ${MAX_TOKENS} \
>     --seed ${SEED}
>     

Am I missing anything?

python version 3.7.6 fairseq version 0.9.0 CUDA Version: 11.0 pyTorch VERSION: 1.7.1

I think the problem might be pytorch version. I met the same problem, downgrade pytorch to 1.4.0, then problem solved.

[ayushbits]

Re-opening after 1 year . @lazerliu Were you able totrain on torch >1.6.0 ?