False positives are over-reported
See original GitHub issueThe motmetrics module seems to be reporting false positive tracks in situations when none exist. I confirmed this by running eval_motchallenge with the 2DMOT2015 ground truth files copied and used as the test results. The metric results are below - most are perfect, as would be expected, but FP can get quite high, and this affects downstream metrics like IDF1 and MOTA.
IDF1 IDP IDR Rcll Prcn GT MT PT ML FP FN IDs FM MOTA MOTP
KITTI-13 90.1% 82.0% 100.0% 100.0% 82.0% 42 42 0 0 167 0 0 0 78.1% -0.000
ADL-Rundle-8 100.0% 100.0% 100.0% 100.0% 100.0% 28 28 0 0 0 0 0 0 100.0% 0.000
Venice-2 100.0% 100.0% 100.0% 100.0% 100.0% 26 26 0 0 0 0 0 0 100.0% 0.000
TUD-Campus 100.0% 100.0% 100.0% 100.0% 100.0% 8 8 0 0 0 0 0 0 100.0% 0.000
KITTI-17 93.2% 87.3% 100.0% 100.0% 87.3% 9 9 0 0 99 0 0 0 85.5% -0.000
ETH-Bahnhof 82.8% 70.6% 100.0% 100.0% 70.6% 171 171 0 0 2255 0 0 0 58.4% 0.000
PETS09-S2L1 98.1% 96.3% 100.0% 100.0% 96.3% 19 19 0 0 174 0 0 0 96.1% 0.000
TUD-Stadtmitte 100.0% 100.0% 100.0% 100.0% 100.0% 10 10 0 0 0 0 0 0 100.0% 0.000
ADL-Rundle-6 100.0% 100.0% 100.0% 100.0% 100.0% 24 24 0 0 0 0 0 0 100.0% 0.000
ETH-Sunnyday 98.9% 97.7% 100.0% 100.0% 97.7% 30 30 0 0 43 0 0 0 97.7% 0.000
ETH-Pedcross2 96.2% 92.7% 100.0% 100.0% 92.7% 133 133 0 0 495 0 0 0 92.1% 0.000
OVERALL 96.1% 92.5% 100.0% 100.0% 92.5% 500 500 0 0 3233 0 0 0 91.9% 0.000
I will look into this a bit and open a PR if I find the cause.
Issue Analytics
- State:
- Created 4 years ago
- Comments:18 (6 by maintainers)
Top Results From Across the Web
WHO did not say PCR test flaw led to overstated COVID-19 ...
CLAIM: The World Health Organization admits that PCR tests to diagnose COVID-19 gave massive false positives, overinflating COVID-19 case ...
Read more >Why False Positives Merit Concern, Too - The New York Times
False negatives are not the only troublesome outcome of a faulty coronavirus test.
Read more >Ohio governor's apparent false positive troubles health ...
Ohio Gov. Mike DeWine's on-again, off-again diagnosis is troubling public health experts, who on Friday expressed fear that the prominent ...
Read more >Epidemiology, Public Health, and the Rhetoric of False Positives
2008, 2009a, 2009b) espouse the viewpoint that “false positive” results are a pervasive problem in epidemiologic research and, coupled with the ...
Read more >Errors in Survey Reporting and Imputation and Their Effects ...
We show that imputations are an important source of survey error, particularly of false positives. Finally, we examine to what extent the data...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
@antonmil. Correct, identical .txt files don’t necessarily mean perfect scores. However identical result matrices fed to the core evaluation algorithms give perfect scores.
py-motmetrics has the same behavior as the official devkit. In summary there are differences in how the GT and Result files are parsed leading to seemingly imperfect scores. Yet the evaluation code gives perfect scores when the GT and Result variables are identical.