Write ingester for TREC 2020 Health Misinformation Track

The collection for the TREC 2020 Health Misinformation Track appears to be common crawl WARCS.

ClueWeb12 is distributed as WARCs also: https://github.com/castorini/anserini/blob/master/src/main/java/io/anserini/collection/ClueWeb12Collection.java

See if we can adapt code there? Or maybe common crawl has their own APIs?

Call this CommonCrawlCollection.

[MXueguang]

An search example using the example topic from https://trec-health-misinfo.github.io

>>> from pyserini.search import SimpleSearcher
>>> searcher = SimpleSearcher('/store/collections/trec-misinfo/warc_index/')
>>> hits = searcher.search('ibuprofen COVID-19')
>>> for i in range(0, 10):
...     print(f'{i+1:2} {hits[i].docid:15} {hits[i].score:.5f}')
...
 1 <urn:uuid:a9a35d9b-989e-4a02-b933-70ad24b9e712> 9.51370
 2 <urn:uuid:d98bdbaa-b5fa-4e39-b3a7-e4d6269219f4> 9.51370
 3 <urn:uuid:c17ed5a6-d96a-471b-b6c7-226192058b8f> 9.47510
 4 <urn:uuid:0b395925-b0b4-45c7-9158-34e91b9e9c64> 9.47280
 5 <urn:uuid:f14fae55-1054-4d61-bda2-adca5ef7b43d> 9.45360
 6 <urn:uuid:edff9bcf-2130-4c96-8aed-fba1c64ffc17> 9.44440
 7 <urn:uuid:998efb65-ca44-4c26-bec5-c0d2c962ca75> 9.41200
 8 <urn:uuid:c620dfe6-c9b8-45a2-bfee-969f5b1151f4> 9.40180
 9 <urn:uuid:9e829fd6-712f-4249-b08c-ae3225e52647> 9.39590
10 <urn:uuid:e8e5c59b-8709-473a-ab0e-e06461b6c77f> 9.38050

Part of hits[0].contents:

'No concrete evidence ibuprofen makes COVID-19 worse: health experts | TheSpec.com No concrete evidence ibuprofen makes COVID-19 worse: health experts Living 09:22 AM by Cassandra Szklarski The Canadian Press TORONTO — Canadian health officials are trying to calm fears that anti-inflammatory drugs such as ibuprofen can worsen COVID-19 symptoms by stressing the lack of concrete evidence. Debate over whether ibuprofen products such as Advil should be bypassed for acetaminophen medications including Tylenol continues to rage among many people confused by conflicting reports spreading online. Alberta's medical health office offered assurances Thursday on Twitter, stating “there is no strong evidence to indicate that ibuprofen could worsen COVID-19 symptoms beyond the usual known side effects.” “Until more information is available, people may wish to take paracetamol/acetaminophen to treat COVID-19 symptoms, unless advised otherwise by their doctor,” said the account, run by public health staff on behalf of chief medical officer Dr. Deena Hinshaw. The executive vice president and chief pharmacy officer of Ontario Pharmacists Association also said Thursday there was not enough evidence to avoid the common painkiller but nevertheless suggested concerned patients use acetaminophen instead. “It's sometimes good to err on the side of caution because we can't disapprove what that statement was,” said Allan Malek, referring to a weekend tweet from France's health ministry that sparked the controversy. "Because there is another alternative — acetaminophen, which are the Tylenol-based products — that would be a good alternative in terms of treating fever and pain that may come along with positive symptoms of COVID-19.

[MarcosFP97]

No worries! Thank you