Getting Nan loss while training

I have a dataset containing 846 images but when start training I am getting there are 1692 images. I have the dataset in PASCAL_VOC format. The JPEGImages folder contains 846 images. On training, I am getting loss:nan. Can you please let me know the reason for the same? Preparing training data... done before filtering, there are 1692 images... after filtering, there are 1692 images... 1692 roidb entries Loading pretrained weights from data/pretrained_model/resnet101_caffe.pth [session 1][epoch 1][iter 0] loss: 6.7142, lr: 1.00e-03 fg/bg=(2/126), time cost: 238.602555 rpn_cls: 0.7190, rpn_box: 1.7119, rcnn_cls: 4.2830, rcnn_box 0.0003 [session 1][epoch 1][iter 100] loss: nan, lr: 1.00e-03 fg/bg=(13/115), time cost: 40.301977 rpn_cls: 0.5280, rpn_box: nan, rcnn_cls: 0.7082, rcnn_box 0.0000 [session 1][epoch 1][iter 200] loss: nan, lr: 1.00e-03 fg/bg=(32/96), time cost: 40.584164 rpn_cls: 0.3966, rpn_box: nan, rcnn_cls: 1.0526, rcnn_box 0.0000 [session 1][epoch 1][iter 300] loss: nan, lr: 1.00e-03 fg/bg=(8/120), time cost: 41.294393 rpn_cls: 0.4398, rpn_box: nan, rcnn_cls: 0.6331, rcnn_box 0.0000 [session 1][epoch 1][iter 400] loss: nan, lr: 1.00e-03 fg/bg=(32/96), time cost: 42.057193 rpn_cls: 0.2161, rpn_box: nan, rcnn_cls: 0.9535, rcnn_box 0.0000 [session 1][epoch 1][iter 500] loss: nan, lr: 1.00e-03 fg/bg=(32/96), time cost: 41.014715 rpn_cls: 0.1673, rpn_box: nan, rcnn_cls: 0.9406, rcnn_box 0.0000 [session 1][epoch 1][iter 600] loss: nan, lr: 1.00e-03 fg/bg=(32/96), time cost: 42.453671 rpn_cls: 0.1687, rpn_box: nan, rcnn_cls: 0.9308, rcnn_box 0.0000

[ashutoshIITK]

@super-wcg Yes, I solved the problem of getting NaN Loss. It was due to the error in the coordinates. The following things were giving NaN loss 1.Coordinates out of the image resolution------------> NaN Loss 2. xmin=xmax-----------> Results in NaN Loss 3. ymin==ymax-----------> Results in Nan Loss 4. The size of bounding box was very small-----------> Results in NaN Loss

For the 4th case, we put a condition that the difference of |xmax -xmin| >= 20 and similarly |ymax- ymin| >=20

I trained the model (For 20 epochs) after fixing all this and didn’t get NaN Loss error.

Thank you.

[swchui]

There are somthing wrong about your dataset. 1.In the “\lib\dataset\pascal_voc.py”, change the" x1 = float(bbox.find(‘xmin’).text) - 1 y1 = float(bbox.find(‘ymin’).text) - 1" to x1 = float(bbox.find(‘xmin’).text) y1 = float(bbox.find(‘ymin’).text) " delete the “-1”. 2. then “rm -rf $your data cache$” Maybe the log(-1) lead to this error.