Add common metrics for information retrieval

There’s a nice summary of common metrics for IR here: https://www.pinecone.io/learn/offline-evaluation/

Although we have many of these as part of trec_eval, it could make sense to have separate metrics like MRR and NDCG@K to give visibility

[ola13]

ICT - Inverse Close Task, e.g. as defined here https://arxiv.org/pdf/1906.00300.pdf

Yes I think what would be more easy for my use-case would be things like Recall / Precision @ n etc, I can take a stab at those then 😃

[ola13]

Hey @cakiki are you still interested in working on this?

I’m thinking about IR metrics in the context of building the retriever for ROOTS - I want a simple way to test the quality of retrievers I’m building and I’m thinking about the following approach right now:

• choose a sample of all documents from ROOTS - similar to https://huggingface.co/datasets/bigscience-data/roots_1e-1 but with two differences - I would be choosing a sample of passages as I define them in my index (I split long documents into multiple passages) and I want the sample to be smaller.
• split each passage from the sample into sentences (potential challenge in some languages)
• create a test set using ICT-style approach - select a random sentence from a passage and use it as a query to retrieve that passage (in my simple test set approach I don’t necessarily want to remove the sentence from the passage)

Then I could be evaluating using huggingface/evaluate IR metrics.

What do you think? I’m also looking for better ideas for creating test sets for ROOTS retrieval 😃