An illegal memory access was encountered

🐛 Bug

I just run the below script with 4 x P100.

PYTHON="/root/miniconda3/bin/python"
CONFIG="./configs/e2e_mask_rcnn_R_50_FPN_1x.yaml"

export NGPUS=4
${PYTHON} -m torch.distributed.launch --nproc_per_node=$NGPUS \
	./tools/train_net.py --config-file $CONFIG

Expected behavior

Here is the error information,

It seems that the first two few iterations are ok. (iter: 0, 20)

Then in the iter 40, the number in the bracket becomes nan. Then I got the error informing me that an illegal memory was encountered.

Environment

I just install all the enviroments follow the instructions

• PyTorch Version 1.0
• Linux 16.04
• Python version: 3.6
• CUDA/cuDNN version: 8.0
• GPU models and configuration: 4 X P100

[PkuRainBow]

@fmassa Thanks for your kind help.

I will update if I have got progress.

[zimenglan-sysu-512]

hi @fmassa i want to add light-head rcnn to train R-50-C4 on COCO dataset, maybe something wrong in my code to implement. i need to check my code. thanks.