Anchor Text for msmarco-document and msmarco-document-v2

Dataset Information:

We have extracted anchor text pointing to documents in MS MARCO (version 1 and version 2) from several Common Crawl snapshots that can be used as additional retrieval features or for the training of models (e.g., in a distant supervision style like DeepCT).

Links to Resources:

• Code and links to the data
• Paper

Dataset ID(s) & supported entities:

• Dataset ID: msmarco-document/anchor-text and msmarco-document-v2/anchor-text

Checklist

Mark each task once completed. All should be checked prior to merging a new dataset.

• Dataset definition (in ir_datasets/datasets/[topid].py)
• Tests (in tests/integration/[topid].py)
• Metadata generated (using ir_datasets generate_metadata command, should appear in ir_datasets/etc/metadata.json)
• Documentation (in ir_datasets/etc/[topid].yaml)
  • Documentation generated in https://github.com/seanmacavaney/ir-datasets.com/
• Downloadable content (in ir_datasets/etc/downloads.json)
  • Download verification action (in .github/workflows/verify_downloads.yml). Only one needed per topid.
  • Any small public files from NIST (or other potentially troublesome files) mirrored in https://github.com/seanmacavaney/irds-mirror/. Mirrored status properly reflected in downloads.json.

Additional comments/concerns/ideas/etc.

I would be happy to help integrate the anchor texts into ir_datasets. I am not sure what a good Dataset ID would be, it can make sense to integrate it as a subset into the existing msmarco-document and msmarco-document-v2 Ids but it might also make sense to have it as independent Ids.

[seanmacavaney]

Excellent, thanks again @mam10eks!

[seanmacavaney]

@mam10eks – can you accept the PR here with the metadata when you get a chance? https://github.com/mam10eks/ir_datasets/pull/1