Unable to reproduce CBGS's results on NuScenes

Comparing to the current master branch, I made two changes in order to fix the NaN training loss.

The first change is described in #46 .

The second change is to add what’s below before line 193 in losses.py

# FIX NaN TARGETS 
target_tensor = torch.where(
    torch.isnan(target_tensor), prediction_tensor, target_tensor
)

Besides, I set

norm_cfg = dict(type='SyncBN', eps=1e-3, momentum=0.01)

in examples/cbgs/configs/nusc_all_vfev3_spmiddleresnetfhd_rpn2_mghead_syncbn.py and

torch.backends.cudnn.benchmark = True

in tools/train.py.

Here are my results on the validation set after training 20 epochs:

car Nusc dist AP@0.5, 1.0, 2.0, 4.0
59.25, 71.87, 77.22, 79.63 mean AP: 0.7199402759604012
truck Nusc dist AP@0.5, 1.0, 2.0, 4.0
17.96, 35.01, 43.00, 47.15 mean AP: 0.357782470829584
construction_vehicle Nusc dist AP@0.5, 1.0, 2.0, 4.0
0.00, 1.28, 6.75, 13.37 mean AP: 0.05348830261303094
bus Nusc dist AP@0.5, 1.0, 2.0, 4.0
23.87, 48.49, 62.98, 66.32 mean AP: 0.5041451034213309
trailer Nusc dist AP@0.5, 1.0, 2.0, 4.0
1.94, 14.27, 30.88, 42.11 mean AP: 0.22300031478924093
barrier Nusc dist AP@0.5, 1.0, 2.0, 4.0
28.06, 48.97, 57.80, 60.27 mean AP: 0.4877375663669212
motorcycle Nusc dist AP@0.5, 1.0, 2.0, 4.0
24.97, 29.29, 30.38, 30.99 mean AP: 0.28906646690838084
bicycle Nusc dist AP@0.5, 1.0, 2.0, 4.0
6.20, 7.36, 7.98, 8.53 mean AP: 0.07516058303100348
pedestrian Nusc dist AP@0.5, 1.0, 2.0, 4.0
62.82, 64.73, 66.83, 69.03 mean AP: 0.658543997130018
traffic_cone Nusc dist AP@0.5, 1.0, 2.0, 4.0
42.10, 44.31, 46.23, 50.65 mean AP: 0.4582346501948114

Overall the mean AP is 38.2, which is much lower than what’s reported.

Can someone point me to what I might have missed? Thanks!

[zxduan90]

Why the mAVE you reproduced is normal? the others are get a larger value, could you share your modified version or config?

Maybe you should pay attention to the arg nsweep in NuscenesDataset(https://github.com/poodarchu/Det3D/blob/b4cb6428b807d8cfe2135b535961d710ff9d41c1/det3d/datasets/nuscenes/nuscenes.py#L35), its default value is 1. However the name should be n_sweep which is the same as the one in config. So I think you may train with only one lidar frame.

[muzi2045]

OK, the CBGS use the global velocity of the annotations. But I am using the ego velocity (in Lidar coords)

# convert velo from global to lidar
for i in range(len(ref_boxes)):
     velo = np.array([*velocity[i], 0.0])
     velo = velo @ np.linalg.inv(e2g_r_mat).T @ np.linalg.inv(
            l2e_r_mat).T
     velocity[i] = velo[:2]
velocity = velocity.reshape(-1,2)

maybe need to check velocity loss compute function…