Add revision to datasets

I’d propose to add the commit hash of the revision to tasks:

from mteb import MTEB
from mteb.abstasks.AbsTaskReranking import AbsTaskReranking
from sentence_transformers import SentenceTransformer


class MindSmallReranking(AbsTaskReranking):
    @property
    def description(self):
        return {
            "name": "MindSmallReranking",
            "hf_hub_name": "mteb/mind_small",
            "description": "Microsoft News Dataset: A Large-Scale English Dataset for News Recommendation Research",
            "reference": "https://www.microsoft.com/en-us/research/uploads/prod/2019/03/nl4se18LinkSO.pdf",
            "type": "Reranking",
            "category": "s2s",
            "eval_splits": ["validation"],
            "eval_langs": ["en"],
            "main_score": "map",
            "revision": "75937953179...",
        }

model = SentenceTransformer("average_word_embeddings_komninos")
evaluation = MTEB(tasks=[MindSmallReranking()])
evaluation.run(model)

This is then fed into load_dataset via revision= & added to the results json file.

This partly addresses https://github.com/embeddings-benchmark/mteb/issues/21

[Muennighoff]

As noted in https://github.com/embeddings-benchmark/mteb/pull/41, this Issue can be closed once a revision field has been added to all MTEB datasets

[Muennighoff]

Yeah there are three versions:

• Revision of the model (Probably entirely on the user side, as we don’t implement models here yet)
• Revision of the dataset
• Version of this library (& its dependencies)

If we don’t have dataset revisions, the current implementation automatically downloads the latest dataset, so it wouldn’t be versioned. As each change to a dataset revision is tracked via a git commit in this repo, using just the version of this library (e.g. a commit string) should be enough. We’ll know all dataset revisions at that commit from the code.