Reproduce 15-1 setup on Pascal VOC

Describe the bug I tried to run the provided pascal VOC script using Apex optimization 01 and everything same as script except i was using a single GPU and hence changed the batch size to 24. But I got the following results

	1-15	16-20	all
Paper	65.12	21.11	54.64
Code results	58.73	21.6	49.7

To Reproduce start=date +%s`

START_DATE=$(date ‘+%Y-%m-%d’)

PORT=$((9000 + RANDOM % 1000)) GPU=0 NB_GPU=1 DATA_ROOT=./data DATASET=voc TASK=15-5s NAME=PLOP METHOD=PLOP BATCH_SIZE=24 INITIAL_EPOCHS=30 EPOCHS=30 OPTIONS=“–checkpoint checkpoints/step/”

RESULTSFILE=results/${START_DATE}${DATASET}${TASK}_${NAME}.csv rm -f ${RESULTSFILE}

CUDA_VISIBLE_DEVICES=${GPU} python3 -m torch.distributed.launch --master_port ${PORT} --nproc_per_node=${NB_GPU} run.py --date ${START_DATE} --data_root ${DATA_ROOT} --overlap --batch_size ${BATCH_SIZE} --dataset ${DATASET} --name ${NAME} --task ${TASK} --step 0 --lr 0.01 --epochs ${INITIAL_EPOCHS} --method ${METHOD} --opt_level O1 ${OPTIONS} for step in 1 2 3 4 5 do CUDA_VISIBLE_DEVICES=${GPU} python3 -m torch.distributed.launch --master_port ${PORT} --nproc_per_node=${NB_GPU} run.py --date ${START_DATE} --data_root ${DATA_ROOT} --overlap --batch_size ${BATCH_SIZE} --dataset ${DATASET} --name ${NAME} --task ${TASK} --step ${step} --lr 0.001 --epochs ${EPOCHS} --method ${METHOD} --opt_level O1 ${OPTIONS} done python3 average_csv.py ${RESULTSFILE}`

[arthurdouillard]

Don’t hesitatee to reopen this issue if you have new findings. Best,

[arthurdouillard]

I think the problem comes from either:

• how gradients are accumulated with multiple GPUs vs a single GPU, maybe you need to tune the learning rate for a single GPU
• does the asyncBN work differently depending on the number of GPUs?