Direct access to all doc_ids

This is something I was expecting to be quite straightforward (or at least better documented in the API) but it doesn’t seem to be. Say I want to gather all doc_ids from a given corpus (for instance, if I want to use a random negative sampler on run time). Currently, this is what I do:

data = ir_datasets.load("msmarco-document/train")
all_doc_ids = list(data.docs._handler.docs_store().lookup.idx())

which is fine, but, from what I can get, this triggers an iteration over all docs in the collection (and is also not very intuitive).

Is there a better way to achieve this?

[seanmacavaney]

Hey @ArthurCamara – quick update on this. Over the past few months I’ve been working on an alternative file format to facilitate doc_id->idx and idx->doc_id lookups, iteration over doc_ids, etc. It also aims to ditch the searchsorted approach for doc_id->idx lookups in favor of an on-disk hash table, since the former requires doc_ids to be padded to the same length (adding considerable size to some lookups) and has an unfavourable access pattern on disk, which makes it a bit slow until everything is loaded into the cache.

Not sure when it’ll be ready for primetime, but just letting you know that a solution to this is in the works.

[ArthurCamara]

That sounds awesome, @seanmacavaney. Thanks for letting me know!