About reproduce CBGS result

Instructions To Reproduce the Issue:

1. what changes you made (git diff) or what code you wrote using the Ego velocity in every annotations, the other config are same.

# convert velo from global to lidar
for i in range(len(ref_boxes)):
    velo = np.array([*velocity[i], 0.0])
    velo = velo @ np.linalg.inv(e2g_r_mat).T @ np.linalg.inv(
          l2e_r_mat).T
    velocity[i] = velo[:2]
    velocity = velocity.reshape(-1,2)

1. what exact command you run:

python3 -m torch.distributed.launch --nproc_per_node=8 ./tools/train.py examples/cbgs/configs/nusc_all_vfev3_spmiddleresnetfhd_rpn2_mghead_syncbn.py --work_dir=/home/ubuntu/Documents/Det3D/trained_model

1. what you observed:

mAP: 0.3719
mATE: 0.3724
mASE: 0.2661
mAOE: 0.9296
mAVE: 1.3655
mAAE: 0.2684
NDS: 0.4023
Eval time: 140.1s

Per-class results:
Object Class	AP	ATE	ASE	AOE	AVE	AAE
car	0.721	0.219	0.158	0.841	1.116	0.230
truck	0.371	0.426	0.198	0.640	1.155	0.307
bus	0.500	0.439	0.174	1.223	2.171	0.431
trailer	0.213	0.687	0.219	0.670	1.371	0.184
construction_vehicle	0.058	0.798	0.481	1.370	0.157	0.372
pedestrian	0.653	0.165	0.287	1.350	0.869	0.439
motorcycle	0.242	0.223	0.243	1.107	3.192	0.153
bicycle	0.043	0.199	0.264	1.111	0.894	0.031
traffic_cone	0.449	0.170	0.348	nan	nan	nan
barrier	0.470	0.398	0.289	0.056	nan	nan
Evaluation nusc: Nusc v1.0-trainval Evaluation
car Nusc dist AP@0.5, 1.0, 2.0, 4.0
59.48, 71.97, 77.40, 79.65 mean AP: 0.7212472062431424
truck Nusc dist AP@0.5, 1.0, 2.0, 4.0
18.48, 36.29, 44.83, 48.91 mean AP: 0.3712787143771077
construction_vehicle Nusc dist AP@0.5, 1.0, 2.0, 4.0
0.00, 2.09, 8.06, 12.96 mean AP: 0.05777510817362395
bus Nusc dist AP@0.5, 1.0, 2.0, 4.0
23.60, 47.30, 62.94, 66.32 mean AP: 0.5003920838518946
trailer Nusc dist AP@0.5, 1.0, 2.0, 4.0
1.66, 13.24, 29.62, 40.49 mean AP: 0.21251682647224052
barrier Nusc dist AP@0.5, 1.0, 2.0, 4.0
26.14, 46.78, 55.95, 59.13 mean AP: 0.4700045657239055
motorcycle Nusc dist AP@0.5, 1.0, 2.0, 4.0
20.39, 24.74, 25.62, 26.05 mean AP: 0.24202605811125658
bicycle Nusc dist AP@0.5, 1.0, 2.0, 4.0
3.93, 4.27, 4.35, 4.58 mean AP: 0.04280152387228541
pedestrian Nusc dist AP@0.5, 1.0, 2.0, 4.0
62.12, 64.40, 66.13, 68.36 mean AP: 0.6525328516852104
traffic_cone Nusc dist AP@0.5, 1.0, 2.0, 4.0
40.81, 43.40, 45.49, 49.80 mean AP: 0.44874109465427564

Unable to reproduce the results in model zoo.

Expected behavior:

the score NDS don’t reach the number in released paper, and the AVE number is abnormal large than others, this reproduced result even worse than pointpillars. Is the loss compute func exist some problems leading to this result?

[turboxin]

hi @muzi2045 @peiyunh, have you tried to train pointpillars on nuscene? And if you do, could you please report the eval result? Thank you a lot~

[poodarchu]

the ground plane is not enabled, and I can’t find it where used this func.

It doesn’t need to use ground plane detection module to reach 52.8 mAP