question-mark

Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

feature request: hits (or accuracy?)

See original GitHub issue

Issue Description

Hi,

@osf9018 mentioned it in #2 but I guess it’s better to create a specific issue.

Motivation

It is often difficult to estimate the total number of relevant document for a query. For example, in Question Answering, if you have a large enough Knowledge Base, you can find the answer to your question in a surprisingly large number of documents that one cannot annotate in advance. Because of this, the relevance of the document is often estimated on-the-go, by checking whether the answer string is in the document retrieved by the system.

Because of this, recall is not an appropriate metric. However, one way to circumvent this is to compute recall “as if” there was only a single relevant document. After averaging over the whole dataset, it corresponds to the proportion of question for which the system retrieved at least one relevant document in top-K. This is what @osf9018 and I call “hits@K” (I can’t remember but I’ve seen it in a paper) and others, such as Karpukhin et al., call “accuracy”. Accuracy is a confusing term IMO.

The request

Would you be interested in implementing or integrating this feature in your library? It might take some renaming but it could be implemented very easily by using the _hits function. It is simply min(1, _hits(qrels, run, k))

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:7 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
AmenRacommented, Feb 2, 2022

Fixed in 0.1.10. Sorry for the inconvenience.

1reaction
AmenRacommented, Feb 1, 2022

Hi,

Added in 0.1.9 as hit_rate. It supports at k as usual.

Closing.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Feature Requests: How your feedback makes Looker better
To be clear, customer feedback is one of several critical inputs to our product planning process, but it's not the only one. Just...
Read more >
GameChanger feature request: SIMPLE OUT button - Reddit
I know GC is all about high levels of detail and accuracy, but I would suggest for a future version a Simple Out...
Read more >
Feature Request - 9to5Mac
Feature Request is a regular 9to5Mac series where authors offer their opinion on how to improve popular hardware or software products.
Read more >
We're Removing Our Feature Request Forum - BrightGauge
Hello everyone! We're going to do something crazy in 2018... Yes, you've read right. We've decided to remove our feature request forums.
Read more >
What's the difference between accuracy and precision? - Asana
Accuracy and precision are both ways to measure results. Accuracy measures how close results are to the true or known value.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found