Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Build Sparse Index for NQ

See original GitHub issue

Hi 😃

First, many thanks for this awesome library!

I wanted to ask - what is the command to build the sparse index over DPR’s data psgs_w100.tsv (i.e., reproduce your wikipedia-dpr index)?

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:10 (5 by maintainers)

github_iconTop GitHub Comments

lintoolcommented, Aug 13, 2021

-storeContents stores the document vector representations in the index, which enable relevance feedback.

Here’s the “model card” for the index:

If you are able to perform retrieval and get the same level of effectiveness, then you should be good.

MXueguangcommented, Aug 8, 2021

@oriram could you double check if you added -storeRaw option? We built the index with raw collection stored in the index. The raw collection should be than 3.5G.

Without storing anything additional things (i.e. wtihout storeRaw storeVector etc.) the pure index will be around 2.5 G

Read more comments on GitHub >

github_iconTop Results From Across the Web

Sparse Indexes — MongoDB Manual
An index that is both sparse and unique prevents collection from having documents with duplicate values for a field but allows multiple documents...
Read more >
SQL Sparse Indexing for Db2 z/OS - IDUG
The inner table is read first applying any local predicates and Db2 building the sparse index. The sparse index result from the inner...
Read more >
Sparse data structures — pandas 0.18.1 documentation
SparseArray is the base layer for all of the sparse indexed data structures. It is a 1-dimensional ndarray-like object storing only values distinct...
Read more >
Sparse indexes - IBM
An SQL sparse index is like a select/omit access path. Both the sparse index and the select/omit logical file contain only keys that...
Read more >
Indexing - UiO
However, sparse indexes must access the data block to see if a record exists ... Can we build a dense, second level index...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found