Using evaluate.py - is it functional?

Since I see no mention of it in anywhere, I’m wondering if evaluate.py is functional?

First I had to modify it to use it at all, then when I tried using it at all on my custom data, I got bad results despite knowing the model is performing quite well.

So to test it, I took 20 ground truth labels, copied them and added fake scores between 0.9 and 1.0. I may be misunderstanding mAP (low number of examples makes the metric weird, but I have ‘perfect’ detection here), but it seems I should be getting reasonably high scores? For cars, I get

Car AP@0.70, 0.70, 0.70:
bbox AP:27.2727, 54.5455, 63.6364
bev  AP:0.0000, 0.0000, 0.0000
3d   AP:0.0000, 0.0000, 0.0000
aos  AP:27.27, 54.55, 63.64

Not even consistent between evaluations. Why 0 BEV and 3D, is it not running on those metrics? I don’t see anything to do with calibration in the eval code, so shouldn’t be an issue with reference frames. I’m lost. Is something wrong with this code?

[jacoblambert]

Since I couldn’t solve this in PCDet repo, I’m resorting to using official eval code. I made some script to transfer whatever your format is to KITTI format, then you can run the cpp code simply: https://github.com/jacoblambert/label_to_kitti_format

[jhultman]

@jacoblambert I think the problem you mention is the same as the one discussed here and here. The rotated IOU algorithm in the eval library cannot handle the case where the orientations are identical (this is a bug of course).