[Question] Recommended KNN/ANN index for large datasets

I would like to use CropResistantHash to quickly find near-duplicates from a large set of reference images.

With other hash functions I would normaly use some kind of approximate nearest neighbor index, such as NMSLib or Annoy. The challenge is that CropResistantHash is variable length and cannot be compared using one of the standard distance functions (Angular, Hamming, Manhattan, …).

Can anyone point me to an alternative solution? How do you use this with large datasets?

[JohannesBuchner]

Some links that could be useful:

• https://dba.stackexchange.com/questions/72134/fast-hamming-distance-queries-in-postgres
• https://stackoverflow.com/questions/14925151/hamming-distance-optimization-for-mysql-or-postgresql
• https://stackoverflow.com/questions/4777070/hamming-distance-on-binary-strings-in-sql
• https://dba.stackexchange.com/questions/72134/fast-hamming-distance-queries-in-postgres

[misotrnka]

I can try that, but I believe that any distance function in a DB would rely on full sequential scan of the table. We are talking about hundreds of millions of rows here, so I think some kind of index is neccessary to narrow the options down a bit. But thank you for the idea, I’ll explore it.