question-mark

Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[BUG] Precision calculation incorrect?

See original GitHub issue

Issue Description

Describe the bug In the below example, I would expect run1 to have a precision of 1.0 and I would expect both run2 and run3 to have precisions of 0.75, as 3 out of 4 returned documents are relevant. Instead the second query returns 0.5, and the third 0.25. Either there is a bug handling empty query results, or I have a naive misunderstanding of precision. Also, run 2 and run 3 are similar, just with different queries returning null results. Please correct me if I’m wrong!

To Reproduce Steps to reproduce the behavior:

qrels_dict = {
    "q_1": {"doc_a": 1},
    "q_2": {"doc_b": 1, "doc_c": 1, "doc_d": 1},
    "q_3": {"doc_e": 1},
    "q_4": {"doc_f": 1},
}

run_dict_1 = {
    "q_1": {"doc_a": 1.0},
    "q_2": {"doc_d": 1.0},
    "q_3": {"doc_e": 1.0},
    "q_4": {"doc_f": 1.0},
}

run_dict_2 = {
    "q_1": {"doc_a": 1.0},
    "q_2": {"doc_d": 1.0},
    "q_3": {},
    "q_4": {"doc_f": 1.0},
}

run_dict_3 = {
    "q_1": {"doc_a": 1.0},
    "q_2": {},
    "q_3": {"doc_e": 1.0},
    "q_4": {"doc_f": 1.0},
}


qrels = Qrels(qrels_dict)
run1 = Run(run_dict_1)
run2 = Run(run_dict_2)
run3 = Run(run_dict_3)

print(evaluate(qrels, run1, ["precision"]))
print(evaluate(qrels, run2, ["precision"]))
print(evaluate(qrels, run3, ["precision"]))

1.0 0.5 0.25

Issue Analytics

  • State:closed
  • Created 2 months ago
  • Comments:6 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
AmenRacommented, Nov 22, 2022

Hi @kaleko,

Thank you very much for the bug report and for providing a working example! numba was not raising a ZeroDivisionError, so I did not spot this issue before. I fixed it in v.0.3.4. Now it works as intended.

Please, consider giving ranx a star if you like it!

0reactions
AmenRacommented, Nov 22, 2022

If you find a theoretically sound explanation, please post it here.

Read more comments on GitHub >

github_iconTop Results From Across the Web

CWE-682: Incorrect Calculation (4.9) - MITRE
The software performs a calculation that generates incorrect or unintended results that are later used in security-critical decisions or resource management.
Read more >
Precision, Accuracy and Total Analytical Error
Precision refers to the magnitude of random errors and the re- producibility of measurements.
Read more >
Precision, Accuracy, and Error - Mometrix Test Preparation
This video tutorial helps explain the basics of Precision, Accuracy, and Error. Get the best test prep review for your exam!
Read more >
Accuracy, Precision, and Error - YouTube
Discussion on the difference between accuracy and precision. Examples of error and percent error calculations.
Read more >
Error Analysis — Precision & Recall | by Toby Chitty - Medium
We can calculate our accuracy measure by adding our TP and TN (predictions we go right) and dividing by the total. But as...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found