About the results

Hi Shaoshuai,

Thanks for your great paper and implementation ~ I have tried your provided checkpoint and it reproduced exactly the same results as what is reported in README.md.

However, when I went through the training process provided in README.md according to your instructions, I got the final results as follows:

bbox AP:97.4753, 89.2702, 88.8074
bev  AP:89.1184, 87.2523, 86.3265
3d   AP:86.1782, 77.1745, 76.6546
aos  AP:97.46, 89.10, 88.54

I got slightly better results (in terms of 3D AP and BEV) if I used RCNN trained at epoch 23 instead of 30 and RPN trained at epoch 200:

bbox AP:97.3265, 89.3636, 88.8258
bev  AP:89.6408, 87.2576, 85.9677
3d   AP:87.3695, 77.6297, 76.8561
aos  AP:97.31, 89.22, 88.59

Both results cannot match up with the pretrained model.

I noticed that the major difference between the model in the provided training pipeline and the official testing pipeline is that the official testing pipeline uses RPN.LOC_XZ_FINE=False. Should I do so if I want to train my model from scratch? Besides, I noticed in the paper that the RCNN stage is trained for 50 epochs. However, in the provided instructions, the epoch number is 30.

Thanks a lot for your reply~

Best, Ken

[Treemann]

@sshaoshuai @kentangSJTU @tengteng95 , Hi~ I used the default configuration to train the offline-augmentation version, and the 3D AP for car (easy) is about 85.2% with (rpn_model: 200 epoch, rcnn_model: 40 epoch). Could you provide your detailed setting for achieving the result you claimed?

[BrickDanD]

@xuntan97 hi,do you solve this problem? I have the same questions.

@kentangSJTU hi, can you help me to solve a problem, i want to get the ap result, so i use the command ‘python eval_rcnn.py --cfg_file cfgs/default.yaml --ckpt …/output/rcnn/default/ckpt/checkpoint_epoch_70.pth --batch_size 6 --eval_mode rcnn --test’. but i don’t get the result like you.

2020-07-07 10:40:20,999 INFO -------------------performance of epoch 70--------------------- 2020-07-07 10:40:20,999 INFO 2020-07-07 10:40:20.999845 2020-07-07 10:40:21,000 INFO final average detections: 6.812 2020-07-07 10:40:21,000 INFO final average rpn_iou refined: 0.000 2020-07-07 10:40:21,000 INFO final average cls acc: 0.000 2020-07-07 10:40:21,000 INFO final average cls acc refined: 0.000 2020-07-07 10:40:21,000 INFO total roi bbox recall(thresh=0.100): 0 / 0 = 0.000000 2020-07-07 10:40:21,000 INFO total roi bbox recall(thresh=0.300): 0 / 0 = 0.000000 2020-07-07 10:40:21,000 INFO total roi bbox recall(thresh=0.500): 0 / 0 = 0.000000 2020-07-07 10:40:21,000 INFO total roi bbox recall(thresh=0.700): 0 / 0 = 0.000000 2020-07-07 10:40:21,000 INFO total roi bbox recall(thresh=0.900): 0 / 0 = 0.000000 2020-07-07 10:40:21,000 INFO total bbox recall(thresh=0.100): 0 / 0 = 0.000000 2020-07-07 10:40:21,000 INFO total bbox recall(thresh=0.300): 0 / 0 = 0.000000 2020-07-07 10:40:21,001 INFO total bbox recall(thresh=0.500): 0 / 0 = 0.000000 2020-07-07 10:40:21,001 INFO total bbox recall(thresh=0.700): 0 / 0 = 0.000000 2020-07-07 10:40:21,001 INFO total bbox recall(thresh=0.900): 0 / 0 = 0.000000 2020-07-07 10:40:21,001 INFO result is saved to: …/output/rcnn/default/eval/epoch_70/test/test_mode

if you can help ，I will be very gratefu