SequenceLib is slower than python-Levenshtien

I was adapting this code for our private use. I noticed that using python-Levenshtein package is 100x better than SequenceMatcher ratio.

Levenshtein.ratio(A, B) gets you the same result. I understand that this library is more for offline benchmarking use, but it doesn’t hurt to be faster 😉 .

btw Can you explain the rationale for the custom cost function for substitutions? Any example on how using it changes outcomes of the path taken.

[sai-prasanna]

Thanks for the reference. We noticed the speed improvements you had done that time (thanks).

BTW I have been cleaning up the code to make it easily usable as a library, making it pip installable with python 3 type annotations etc. If you plan to make this a library published to pypi, will give a pull request. Or I would consider publishing as a separate library.

One another side note, want to understand whether homophones can be added as another static rule for subsitutions, as they are among a easy class of errors people make …

P.S. I think I we can consider moving the Damerau Levenshtein code to cython for speed improvements. But thats a separate task in itself. Realized how slow python is compared to cython in python-Levenshtein.

[chrisjbryant]

Aha nice.

I was actually originally using the Damerau-Levenshtein code bundled with ERRANT (rdlextra.py) to do the character alignment, but found SequenceMatcher to be a lot faster. If python-Levenshtein is faster still, then that might definitely be something worth upgrading.

As for the rationale behind the custom substitution cost, it’s easiest to refer you to the alignment paper, particularly section 3.2 and Table 2.