Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

TREC DL 2021

See original GitHub issue

Dataset Information:

Another year of the DL track. From the current description, it looks like it will be the same format as prior years.

Links to Resources:

Dataset ID(s):

(assuming collections remain the same)

msmarco-document/trec-dl-2021 - queries, docs, scoreddocs, (eventually) qrels
msmarco-passage/trec-dl-2021 - queries, docs, scoreddocs, (eventually) qrels

Supported Entities

docs
queries
qrels
scoreddocs
docpairs

Additional comments/concerns/ideas/etc.

Still waiting for the release of datasets & potentially new information.

Issue Analytics

State:
Created 3 years ago
Comments:7

Top GitHub Comments

1reaction

seanmacavaneycommented, Jul 14, 2021

TODO: enable HTTP range requests for azure downloads via an HTTP header to be able to recover from network interruptions: https://github.com/microsoft/msmarco/issues/7#issuecomment-880104882

0reactions

seanmacavaneycommented, Jul 25, 2021

Identified here: https://github.com/allenai/ir_datasets/actions/runs/1064186261

Looks like there was a problem with the passage qrels and they were re-uploaded: https://github.com/microsoft/msmarco/commit/41b3a684ed8ebd4e753250c3687547a77c62e7dd

Since it was only removing duplicate IDs in the file, it should not have an effect on downstream evaluations. It may have an effect on systems that use the qrels for another purpose, e.g., training.

Need to fix hashes, sizes, tests.

Should tests look for such cases and warn if it occurs?

Top Results From Across the Web

TREC 2021 Deep Learning Track Guidelines | msmarco

In 2021, the track will continue to have the same tasks (document ranking and passage ranking) and goals. Similar to the previous year,...

Overview of the TREC 2021 deep learning track - Microsoft

This is the third year of the TREC Deep Learning track. As in previous years, we leverage the MS MARCO datasets that made...

OVERVIEW OF THE TREC 2021 DEEP LEARNING TRACK

At TREC 2021, we hosted the third TREC Deep Learning Track continuing our focus on benchmarking ad hoc retrieval methods in the large-data...

University of Glasgow Terrier Team at the TREC 2021 Deep ...

smac.pub: trec2021-dl. Abstract: This paper describes our submission to the document ranking and passage ranking tasks of the TREC 2021 Deep Learning Track....

[2205.11245] PASH at TREC 2021 Deep Learning Track - arXiv

This paper describes the PASH participation in TREC 2021 Deep Learning Track. In the recall stage, we adopt a scheme combining sparse and...