Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Lower results when evaluating released BEVDet checkpoint

See original GitHub issue

Hello, I have tried to evaluate released BEVDet checkpoint as-is on my setup, but I get

mAP: 0.2751                                                                                                                                                                                   
mATE: 0.7179
mASE: 0.2738
mAOE: 0.5512
mAVE: 0.8747
mAAE: 0.2205
NDS: 0.3737
Eval time: 107.4s

Per-class results:
Object Class    AP      ATE     ASE     AOE     AVE     AAE
car     0.441   0.631   0.167   0.131   1.037   0.254
truck   0.197   0.757   0.225   0.125   0.828   0.227
bus     0.283   0.680   0.185   0.139   1.895   0.350
trailer 0.132   1.053   0.224   0.463   0.547   0.068
construction_vehicle    0.066   0.795   0.484   1.174   0.095   0.358
pedestrian      0.301   0.788   0.305   1.320   0.848   0.412
motorcycle      0.235   0.704   0.262   0.612   1.437   0.090
bicycle 0.182   0.607   0.265   0.875   0.310   0.006
traffic_cone    0.445   0.616   0.333   nan     nan     nan
barrier 0.468   0.547   0.287   0.122   nan     nan

which is lower than the expected 30.8/40.4 mAP/NDS.

I am using A6000 GPUs, torch 1.10.1, cudatoolkit 11.3. Do you know what might be the issue?

I find that I have the exact same numbers as #15 @BoLang615, but I believe I am using the latest version. I would appreciate any pointers for this.

Thank you!

Issue Analytics

State:
Created a year ago
Comments:11 (11 by maintainers)

Top GitHub Comments

1reaction

HuangJunJie2017commented, Jul 16, 2022

@Divadi nice job! thank you so much for your information!

0reactions

Divadicommented, Jul 16, 2022

@HuangJunJie2017 Whew… I think I found the issue; I had Pillow 9.2.0 installed, probably causing some of the operations in image transforms (loading.py) to be slightly different from your Pillow 8.4.0. As a consequence, your loaded images’ differences with mine looked like this: After downgrading to Pillow 8.4.0, the difference is nil:

Updated results:

mAP: 0.3082
mATE: 0.6648
mASE: 0.2729
mAOE: 0.5330
mAVE: 0.8287
mAAE: 0.2052
NDS: 0.4036
Eval time: 98.1s

Per-class results:
Object Class    AP      ATE     ASE     AOE     AVE     AAE
car     0.508   0.535   0.159   0.127   0.947   0.232
truck   0.222   0.671   0.216   0.123   0.834   0.220
bus     0.311   0.760   0.195   0.086   1.592   0.301
trailer 0.150   0.987   0.229   0.443   0.518   0.054
construction_vehicle    0.073   0.720   0.482   1.093   0.103   0.342
pedestrian      0.336   0.738   0.301   1.326   0.861   0.409
motorcycle      0.262   0.704   0.262   0.595   1.450   0.075
bicycle 0.213   0.525   0.270   0.885   0.325   0.009
traffic_cone    0.506   0.518   0.331   nan     nan     nan
barrier 0.502   0.490   0.284   0.119   nan     nan

Thank you for your help!