HC4

Dataset Information:

The HC4 collection that is accepted at ECIR 2022

<brief description>

Links to Resources:

https://github.com/hltcoe/HC4/tree/main/resouces/hc4

Dataset ID(s) & supported entities:

• Dataset ID: hc4/{language id: zh, fa, ru}/{train, dev, test}
• Will have {Chinese, Farsi, Russian} documents, English queries(title/description/narrative), an English report associated with each topic and qrels.

Checklist

Mark each task once completed. All should be checked prior to merging a new dataset.

• Dataset definition (in ir_datasets/datasets/[topid].py)
• Tests (in tests/integration/[topid].py)
• Metadata generated (using ir_datasets generate_metadata command, should appear in ir_datasets/etc/metadata.json)
• Documentation (in ir_datasets/etc/[topid].yaml)
  • Documentation generated in https://github.com/seanmacavaney/ir-datasets.com/
• Downloadable content (in ir_datasets/etc/downloads.json)
  • Download verification action (in .github/workflows/verify_downloads.yml). Only one needed per topid.
  • Any small public files from NIST (or other potentially troublesome files) mirrored in https://github.com/seanmacavaney/irds-mirror/. Mirrored status properly reflected in downloads.json.

Additional comments/concerns/ideas/etc.

The document id, qrels, and topics will be distributed through a public github repository. Users need to download the actual documents through Common Crawl. Script for downloading and validating will be provided along with the doc ids.

The structure will be very similar to the future NeuCLIR collection. Whether these two collections will be distributed through the same repository is TBD.

[seanmacavaney]

Yeah, that’s why I was saying in this edge case, the user would have to take the intersection of qids in hc4/fa/train and hc4/zh/train – leaving just Topic 3. They’d also have to filter the qrels, true.

[eugene-yang]

Thanks, Sean. This makes sense. The top-level should be just placeholder. At this point, I don’t think combining all three languages makes sense. Even though there are some topics that span across languages (i.e. same title and description), but the narratives are different. So they should be considered different queries.

Reports range from 1 to 5 paragraphs. Conceptually, they are written by the analysts prior to the search to reflect some background of the information need.