Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

unstable mAP performance of Retinanet

See original GitHub issue

Thanks for your error report and we appreciate it a lot.

Checklist

I have searched related issues but cannot get the expected help.
The bug has not been fixed in the latest version.

Describe the bug I trained the Retinanet based on Resnet50 for several times, but each time the mAP is different and fluctuate significantly. Reproduction

What command or script did you run?

./tools/dist_train.sh configs/retinanet_r50_fpn_1x.py 4 --validate --seed 47

Did you make any modifications on the code or config? Did you understand what you have modified? `# model settings model = dict( type=‘RetinaNet’, pretrained=‘torchvision://resnet50’, backbone=dict( type=‘ResNet’, depth=50, num_stages=4, out_indices=(0, 1, 2, 3), frozen_stages=1, style=‘pytorch’), neck=dict( type=‘FPN’, in_channels=[256, 512, 1024, 2048], out_channels=256, start_level=1, add_extra_convs=True, num_outs=5), bbox_head=dict( type=‘RetinaHead’, num_classes=21, in_channels=256, stacked_convs=4, feat_channels=256, octave_base_scale=4, scales_per_octave=3, anchor_ratios=[0.5, 1.0, 2.0], anchor_strides=[8, 16, 32, 64, 128], target_means=[.0, .0, .0, .0], target_stds=[1.0, 1.0, 1.0, 1.0], loss_cls=dict( type=‘FocalLoss’, use_sigmoid=True, gamma=2.0, alpha=0.25, loss_weight=1.0), loss_bbox=dict(type=‘SmoothL1Loss’, beta=0.11, loss_weight=1.0)))

training and testing settings

train_cfg = dict( assigner=dict( type=‘MaxIoUAssigner’, pos_iou_thr=0.5, neg_iou_thr=0.4, min_pos_iou=0, ignore_iof_thr=-1), allowed_border=-1, pos_weight=-1, debug=False) test_cfg = dict( nms_pre=1000, min_bbox_size=0, score_thr=0.05, nms=dict(type=‘nms’, iou_thr=0.5), max_per_img=100)

dataset settings

dataset_type = ‘XMLDataset’ data_root = ‘/data/VOC2007/’ img_norm_cfg = dict( mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True) train_pipeline = [ dict(type=‘LoadImageFromFile’), dict(type=‘LoadAnnotations’, with_bbox=True), dict(type=‘Resize’, img_scale=(1333,800),keep_ratio=True), dict(type=‘RandomFlip’, flip_ratio=0.5), dict(type=‘Normalize’, **img_norm_cfg), dict(type=‘Pad’, size_divisor=32), dict(type=‘DefaultFormatBundle’), dict(type=‘Collect’, keys=[‘img’, ‘gt_bboxes’, ‘gt_labels’]), ] test_pipeline = [ dict(type=‘LoadImageFromFile’), dict( type=‘MultiScaleFlipAug’, img_scale=(1333, 800), flip=False, transforms=[ dict(type=‘Resize’, keep_ratio=True), dict(type=‘RandomFlip’), dict(type=‘Normalize’, **img_norm_cfg), dict(type=‘Pad’, size_divisor=32), dict(type=‘ImageToTensor’, keys=[‘img’]), dict(type=‘Collect’, keys=[‘img’]), ]) ] data = dict( imgs_per_gpu=2, workers_per_gpu=2, train=dict( type=dataset_type, ann_file=data_root + ‘ImageSets/Main/trainval.txt’, img_prefix=data_root , pipeline=train_pipeline), val=dict( type=dataset_type, ann_file=data_root + ‘ImageSets/Main/test.txt’, img_prefix=data_root, pipeline=test_pipeline), test=dict( type=dataset_type, ann_file=data_root + ‘ImageSets/Main/test.txt’, img_prefix=data_root, pipeline=test_pipeline))

optimizer

optimizer = dict(type=‘SGD’, lr=0.005, momentum=0.9, weight_decay=0.0001) optimizer_config = dict(grad_clip=dict(max_norm=35, norm_type=2))

learning policy

lr_config = dict( policy=‘step’, warmup=‘linear’, warmup_iters=500, warmup_ratio=1.0 / 3, step=[8, 11]) checkpoint_config = dict(interval=1)

yapf:disable

log_config = dict( interval=50, hooks=[ dict(type=‘TextLoggerHook’), # dict(type=‘TensorboardLoggerHook’) ])

yapf:enable

runtime settings

total_epochs = 12 device_ids = range(8) dist_params = dict(backend=‘nccl’) log_level = ‘INFO’ work_dir = ‘./work_dirs/retinanet_r50_fpn_1x’ load_from = None resume_from = None workflow = [(‘train’, 1)] ` 3. What dataset did you use? Pascal VOC 2007 Environment

OS: Ubuntu 16.04
GCC gcc version 5.4.0
PyTorch version 1.3.0
How you installed PyTorch conda
GPU model 2080Ti
CUDA and CUDNN version CUDA 10.1.243 CUDNN CUDNN_MAJOR 7
[optional] Other information that may be related (such as $PATH, $LD_LIBRARY_PATH, $PYTHONPATH, etc.)

Error traceback If applicable, paste the error trackback here.

**first time**
+-------------+------+-------+--------+-----------+-------+
| class       | gts  | dets  | recall | precision | ap    |
+-------------+------+-------+--------+-----------+-------+
| aeroplane   | 285  | 7036  | 0.947  | 0.038     | 0.725 |
| bicycle     | 337  | 9368  | 0.976  | 0.035     | 0.768 |
| bird        | 459  | 8858  | 0.928  | 0.048     | 0.732 |
| boat        | 263  | 17412 | 0.935  | 0.014     | 0.521 |
| bottle      | 469  | 18215 | 0.885  | 0.023     | 0.607 |
| bus         | 213  | 8581  | 0.962  | 0.024     | 0.734 |
| car         | 1201 | 15242 | 0.975  | 0.078     | 0.858 |
| cat         | 358  | 7647  | 0.978  | 0.046     | 0.846 |
| chair       | 756  | 26292 | 0.922  | 0.027     | 0.493 |
| cow         | 244  | 7733  | 0.984  | 0.031     | 0.443 |
| diningtable | 206  | 15885 | 0.942  | 0.012     | 0.518 |
| dog         | 489  | 10877 | 0.992  | 0.045     | 0.788 |
| horse       | 348  | 9337  | 0.977  | 0.037     | 0.745 |
| motorbike   | 325  | 10406 | 0.948  | 0.030     | 0.722 |
| person      | 4528 | 51407 | 0.971  | 0.087     | 0.841 |
| pottedplant | 480  | 17407 | 0.865  | 0.024     | 0.424 |
| sheep       | 242  | 7362  | 0.909  | 0.030     | 0.501 |
| sofa        | 239  | 12044 | 0.979  | 0.020     | 0.587 |
| train       | 282  | 7786  | 0.947  | 0.034     | 0.750 |
| tvmonitor   | 308  | 11770 | 0.922  | 0.024     | 0.740 |
+-------------+------+-------+--------+-----------+-------+
| mAP         |      |       |        |           | 0.667 |
+-------------+------+-------+--------+-----------+-------+

**second time** 
+-------------+------+-------+--------+-----------+-------+
| class       | gts  | dets  | recall | precision | ap    |
+-------------+------+-------+--------+-----------+-------+
| aeroplane   | 285  | 7040  | 0.940  | 0.038     | 0.715 |
| bicycle     | 337  | 9147  | 0.973  | 0.036     | 0.769 |
| bird        | 459  | 8569  | 0.922  | 0.050     | 0.734 |
| boat        | 263  | 15936 | 0.909  | 0.015     | 0.499 |
| bottle      | 469  | 16763 | 0.885  | 0.025     | 0.626 |
| bus         | 213  | 8504  | 0.944  | 0.024     | 0.690 |
| car         | 1201 | 18553 | 0.977  | 0.064     | 0.862 |
| cat         | 358  | 7863  | 0.980  | 0.045     | 0.845 |
| chair       | 756  | 27806 | 0.925  | 0.026     | 0.518 |
| cow         | 244  | 8079  | 0.980  | 0.030     | 0.360 |
| diningtable | 206  | 16205 | 0.937  | 0.012     | 0.496 |
| dog         | 489  | 10666 | 0.994  | 0.046     | 0.777 |
| horse       | 348  | 9191  | 0.980  | 0.037     | 0.721 |
| motorbike   | 325  | 10285 | 0.966  | 0.031     | 0.766 |
| person      | 4528 | 50014 | 0.968  | 0.089     | 0.837 |
| pottedplant | 480  | 18394 | 0.879  | 0.023     | 0.446 |
| sheep       | 242  | 8410  | 0.917  | 0.027     | 0.438 |
| sofa        | 239  | 11443 | 0.979  | 0.021     | 0.587 |
| train       | 282  | 7668  | 0.950  | 0.035     | 0.659 |
| tvmonitor   | 308  | 11817 | 0.912  | 0.024     | 0.712 |
+-------------+------+-------+--------+-----------+-------+
| mAP         |      |       |        |           | 0.653 |
+-------------+------+-------+--------+-----------+-------+

**the third time**
+-------------+------+-------+--------+-----------+-------+
| class       | gts  | dets  | recall | precision | ap    |
+-------------+------+-------+--------+-----------+-------+
| aeroplane   | 285  | 7150  | 0.930  | 0.037     | 0.692 |
| bicycle     | 337  | 10627 | 0.976  | 0.031     | 0.770 |
| bird        | 459  | 8256  | 0.930  | 0.052     | 0.749 |
| boat        | 263  | 15359 | 0.932  | 0.016     | 0.534 |
| bottle      | 469  | 17853 | 0.889  | 0.023     | 0.593 |
| bus         | 213  | 8596  | 0.962  | 0.024     | 0.726 |
| car         | 1201 | 17710 | 0.972  | 0.067     | 0.868 |
| cat         | 358  | 7036  | 0.983  | 0.050     | 0.835 |
| chair       | 756  | 25050 | 0.930  | 0.029     | 0.525 |
| cow         | 244  | 7772  | 0.980  | 0.031     | 0.410 |
| diningtable | 206  | 14617 | 0.922  | 0.013     | 0.504 |
| dog         | 489  | 10666 | 0.992  | 0.046     | 0.797 |
| horse       | 348  | 9686  | 0.977  | 0.035     | 0.745 |
| motorbike   | 325  | 11117 | 0.969  | 0.028     | 0.763 |
| person      | 4528 | 49964 | 0.971  | 0.089     | 0.841 |
| pottedplant | 480  | 16778 | 0.873  | 0.025     | 0.447 |
| sheep       | 242  | 7203  | 0.917  | 0.031     | 0.537 |
| sofa        | 239  | 9468  | 0.975  | 0.025     | 0.576 |
| train       | 282  | 7947  | 0.943  | 0.034     | 0.737 |
| tvmonitor   | 308  | 9819  | 0.919  | 0.029     | 0.727 |
+-------------+------+-------+--------+-----------+-------+
| mAP         |      |       |        |           | 0.669 |
+-------------+------+-------+--------+-----------+-------+

**the fourth time**
+-------------+------+-------+--------+-----------+-------+
| class       | gts  | dets  | recall | precision | ap    |
+-------------+------+-------+--------+-----------+-------+
| aeroplane   | 285  | 7965  | 0.951  | 0.034     | 0.731 |
| bicycle     | 337  | 9924  | 0.973  | 0.033     | 0.776 |
| bird        | 459  | 8592  | 0.943  | 0.051     | 0.732 |
| boat        | 263  | 16409 | 0.932  | 0.015     | 0.443 |
| bottle      | 469  | 17304 | 0.885  | 0.024     | 0.624 |
| bus         | 213  | 10056 | 0.962  | 0.020     | 0.659 |
| car         | 1201 | 16829 | 0.973  | 0.071     | 0.852 |
| cat         | 358  | 8436  | 0.983  | 0.042     | 0.826 |
| chair       | 756  | 27670 | 0.917  | 0.025     | 0.489 |
| cow         | 244  | 8232  | 0.980  | 0.029     | 0.295 |
| diningtable | 206  | 16879 | 0.937  | 0.011     | 0.532 |
| dog         | 489  | 11085 | 0.996  | 0.044     | 0.764 |
| horse       | 348  | 9687  | 0.971  | 0.035     | 0.707 |
| motorbike   | 325  | 9402  | 0.951  | 0.033     | 0.748 |
| person      | 4528 | 50815 | 0.969  | 0.088     | 0.835 |
| pottedplant | 480  | 16824 | 0.858  | 0.025     | 0.437 |
| sheep       | 242  | 7120  | 0.909  | 0.031     | 0.395 |
| sofa        | 239  | 12311 | 0.975  | 0.019     | 0.580 |
| train       | 282  | 8391  | 0.954  | 0.032     | 0.649 |
| tvmonitor   | 308  | 12194 | 0.922  | 0.023     | 0.725 |
+-------------+------+-------+--------+-----------+-------+
| mAP         |      |       |        |           | 0.640 |
+-------------+------+-------+--------+-----------+-------+

Bug fix Do not find the bug.

Issue Analytics

State:
Created 4 years ago
Comments:11 (1 by maintainers)

Top GitHub Comments

1reaction

leochangzliaocommented, Nov 29, 2019

Finally, I solved this problem and got stable results. Thanks, this issue will be closed.

1reaction

leochangzliaocommented, Nov 22, 2019

I have fixed the seed to 47, but the mAP still fluctuates greatly ( from 0.640 to 0.669). It is a dilemma for me when I want to do some ablation studies by modifying some parameters because of unstable mAP performance.

Any suggestions for making the mAP more stable for small datasets? Thanks. @hellock