Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Poor performance, testing on waymo dataset using kitti pretrained PointRCNN model

See original GitHub issue

Hi, thanks for sharing your nice work. I just try testing on waymo dataset using kitti pretrained PointRCNN model and got poor result.

Change I made in waymo_dataset.yaml:

POINT_FEATURE_ENCODING: {
    encoding_type: absolute_coordinates_encoding,
    used_feature_list: ['x', 'y', 'z', 'intensity'],
    src_feature_list: ['x', 'y', 'z', 'intensity'],
}

My pointrcnn.yaml is copied from kitti_models/pointrcnn.yaml with modification of vehicle mean_size:

POINT_HEAD:
        NAME: PointHeadBox
        CLS_FC: [256, 256]
        REG_FC: [256, 256]
        CLASS_AGNOSTIC: False
        USE_POINT_FEATURES_BEFORE_FUSION: False
        TARGET_CONFIG:
            GT_EXTRA_WIDTH: [0.2, 0.2, 0.2]
            BOX_CODER: PointResidualCoder
            BOX_CODER_CONFIG: {
                'use_mean_size': True,
                'mean_size': [
                    [4.1, 2.1, 1.7],
                    [0.8, 0.6, 1.73],
                    [1.76, 0.6, 1.73]
                ]
            }

My result:

2021-07-23 04:25:49,945   INFO  *************** EPOCH 7870 EVALUATION *****************
2021-07-23 04:26:26,205   INFO  *************** Performance of EPOCH 7870 *****************
2021-07-23 04:26:26,205   INFO  Generate label finished(sec_per_example: 0.0509 second).
2021-07-23 04:26:26,205   INFO  recall_roi_0.3: 0.086557
2021-07-23 04:26:26,206   INFO  recall_rcnn_0.3: 0.086850
2021-07-23 04:26:26,206   INFO  recall_roi_0.5: 0.063807
2021-07-23 04:26:26,206   INFO  recall_rcnn_0.5: 0.067835
2021-07-23 04:26:26,206   INFO  recall_roi_0.7: 0.024971
2021-07-23 04:26:26,206   INFO  recall_rcnn_0.7: 0.037225
2021-07-23 04:26:26,207   INFO  Average predicted number of objects(713 samples): 4.381
2021-07-23 04:26:47,717   INFO
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/AP: 0.0498
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/APH: 0.0468
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/AP: 0.0336
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/APH: 0.0319
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/AP: 0.0103
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/APH: 0.0077
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/AP: 0.0097
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/APH: 0.0072
OBJECT_TYPE_TYPE_SIGN_LEVEL_1/AP: 0.0000
OBJECT_TYPE_TYPE_SIGN_LEVEL_1/APH: 0.0000
OBJECT_TYPE_TYPE_SIGN_LEVEL_2/AP: 0.0000
OBJECT_TYPE_TYPE_SIGN_LEVEL_2/APH: 0.0000
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/AP: 0.0017
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/APH: 0.0017
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/AP: 0.0017
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/APH: 0.0016

I encountered some dependency issue so I installed waymo-open-dataset-tf-2-1-0==1.2.0, instead of waymo-open-dataset-tf-2-0 -0==1.2.0 I am not sure if that is the cause of the poor result (it seems to me not). Do you have any idea what leads to the poor result? Thank you very much.

Issue Analytics

State:
Created 2 years ago
Comments:10 (3 by maintainers)

Top GitHub Comments

2reactions

MartinHahnercommented, Jul 26, 2021

LiDAR models do not transfer well from one dataset to another one. Here is a good read from @ltriess giving an overview on the topic.

1reaction

Shuntw6096commented, Oct 6, 2021

Hi, I have ran kitti -> waymo experiment of Train in Germany, Test in The USA: Making 3D Object Detectors Generalize, I find that a setting of pointrcnn is different from OpenPCDet and paper’s code. https://github.com/cxy1997/3D_adapt_auto_driving/blob/master/pointrcnn/tools/cfgs/default.yaml#L28 So, I change kitti dataset setting https://github.com/open-mmlab/OpenPCDet/blob/c9d31d393acaae34d74cb03bd6ccff9976d3d1f3/tools/cfgs/dataset_configs/kitti_dataset.yaml#L49-L50 to

    used_feature_list: ['x', 'y', 'z',],
    src_feature_list: ['x', 'y', 'z',],

and use paper’s code to translate waymo dataset to kitti format, and get a different result, perhaps the result is more reasonable, but has a gap with paper

2021-10-03 18:58:46,535   INFO  *************** EPOCH 71 EVALUATION *****************
2021-10-03 19:04:41,980   INFO  *************** Performance of EPOCH 71 *****************
2021-10-03 19:04:41,980   INFO  Generate label finished(sec_per_example: 0.1394 second).
2021-10-03 19:04:41,980   INFO  recall_roi_0.3: 0.592279
2021-10-03 19:04:41,981   INFO  recall_rcnn_0.3: 0.597099
2021-10-03 19:04:41,981   INFO  recall_roi_0.5: 0.489929
2021-10-03 19:04:41,981   INFO  recall_rcnn_0.5: 0.538952
2021-10-03 19:04:41,981   INFO  recall_roi_0.7: 0.130628
2021-10-03 19:04:41,981   INFO  recall_rcnn_0.7: 0.286089
2021-10-03 19:04:41,982   INFO  Average predicted number of objects(2549 samples): 6.011
2021-10-03 19:04:46,425   INFO  Car AP@0.70, 0.70, 0.70:
bbox AP:36.8763, 38.8078, 36.5180
bev  AP:39.4154, 41.3404, 37.1790
3d   AP:11.1114, 11.4454, 12.1022
aos  AP:36.64, 38.55, 36.30
Car AP_R40@0.70, 0.70, 0.70:
bbox AP:33.8597, 35.4379, 35.6711
bev  AP:37.5705, 38.7479, 36.7095
3d   AP:9.0654, 10.1663, 10.1325
aos  AP:33.62, 35.19, 35.37
Car AP@0.70, 0.50, 0.50:
bbox AP:36.8763, 38.8078, 36.5180
bev  AP:70.8551, 70.4873, 69.7542
3d   AP:65.7294, 61.2301, 60.8189
aos  AP:36.64, 38.55, 36.30
Car AP_R40@0.70, 0.50, 0.50:
bbox AP:33.8597, 35.4379, 35.6711
bev  AP:73.8100, 71.9384, 67.7279
3d   AP:64.7294, 63.3294, 59.4869
aos  AP:33.62, 35.19, 35.37

and run stat norm code between kitti and waymo, re-train pointrcnn with kitti after stat norm, the result is similar to the paper, but also has a gap with paper.

2021-10-04 17:07:55,597   INFO  *************** EPOCH 80 EVALUATION *****************
2021-10-04 17:14:03,482   INFO  *************** Performance of EPOCH 80 *****************
2021-10-04 17:14:03,483   INFO  Generate label finished(sec_per_example: 0.1443 second).
2021-10-04 17:14:03,483   INFO  recall_roi_0.3: 0.626237
2021-10-04 17:14:03,483   INFO  recall_rcnn_0.3: 0.628432
2021-10-04 17:14:03,483   INFO  recall_roi_0.5: 0.581303
2021-10-04 17:14:03,483   INFO  recall_rcnn_0.5: 0.593398
2021-10-04 17:14:03,483   INFO  recall_roi_0.7: 0.415340
2021-10-04 17:14:03,483   INFO  recall_rcnn_0.7: 0.479685
2021-10-04 17:14:03,484   INFO  Average predicted number of objects(2549 samples): 5.109
2021-10-04 17:14:07,974   INFO  Car AP@0.70, 0.70, 0.70:
bbox AP:51.5635, 53.4044, 48.2356
bev  AP:67.8604, 61.6724, 61.0304
3d   AP:47.4996, 48.2412, 47.3004
aos  AP:51.19, 52.99, 47.88
Car AP_R40@0.70, 0.70, 0.70:
bbox AP:50.4320, 50.7123, 47.8743
bev  AP:65.9226, 64.2689, 59.9811
3d   AP:47.9117, 48.7304, 43.9152
aos  AP:50.05, 50.33, 47.46
Car AP@0.70, 0.50, 0.50:
bbox AP:51.5635, 53.4044, 48.2356
bev  AP:71.0432, 70.7979, 62.9648
3d   AP:70.0923, 70.1391, 62.5190
aos  AP:51.19, 52.99, 47.88
Car AP_R40@0.70, 0.50, 0.50:
bbox AP:50.4320, 50.7123, 47.8743
bev  AP:70.8459, 68.6688, 64.0247
3d   AP:70.0051, 67.9982, 63.4958
aos  AP:50.05, 50.33, 47.46

By the way, I check waymo dataset after tranlated to kitti foramt with kitti_object_vis , but it seems that the location of the boxes is somewhat deviated, not sure whether this will affect the results. Imgur Imgur