Easy computation of all duplicates?

Thanks for this library!

One question: Is there an easy way to get all pairs from a LSH? In other words, instead of a query for a single minhash, I need all pairs of records inside a same bucket.

I tried to dive into the code, but I didn’t understand very well how hashranges/hashtables work. What I need is similar to candidate_duplicates of this other implementation: https://mattilyra.github.io/2017/05/23/document-deduplication-with-lsh.html

[fjsj]

@sameertikoo that’s called the transitivity problem on Record Linkage. Check those slides, page 32. Quick solution is to make the pairs a graph and compute the connected components. Use networkx for that (or networkit if your graph has millions of edges). For an introduction on Record Linkage and the Transitivity problem, I have a talk on deduplication with Python.

[ekzhu]

@sameertikoo the expert has spoken