Store MinHashes for later queries

Hi - I am using MinHashLSH for querying duplicate documents. This is exact flow of my use case.

1. I have around 100K documents.
2. when i open any document, it should show all other documents which are 90% match.

For this, i am first calculating minhashes for each document and then adding it below LSH object.

MinHashLSH(num_perm=perms,threshold=0.9) Issue is - this min hashes creation process itself takes around 470 sec. So i can’t create it every time i query new document.

So I am planning to store this LSH object in disk for future queries and update it may be once in a day/week.

Did any one try this before or know any better way to handle this type of use case?

Thank You.

[aastafiev]

For right deleting document from MinHashLSH index you should use “remove” method from MinHashLSH. See, documentation on API https://ekzhu.github.io/datasketch/documentation.html#minhash-lsh

[aastafiev]

Hi,

You should use pickling. See https://ekzhu.github.io/datasketch/lsh.html#connecting-to-existing-minhash-lsh