Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Detect source language with langdetect package

See original GitHub issue

The langdetect has worked well for me in the past for language detection problems. How would you feel about allowing users to pass 'auto' as an option for source? I could see some pros and cons:

Pros

Users don’t need to be able to recognize a language to translate
Eliminates pre-classification of languages if your dataset contains multiple languages

Cons

Adds another dependency
langdetect detects these 55 languages only

I’m a little new to open source but I would love to contribute 🙂 Of course, if you feel this doesn’t fit this package’s mission that’s totally understandable.

Issue Analytics

State:
Created 2 years ago
Comments:5 (3 by maintainers)

Top GitHub Comments

1reaction

awalker88commented, Apr 26, 2021

Those are some good points, I agree it would be confusing to have the library detect a language but not translate it. I’ll take a look into writing something that could potentially put into the user guide.

0reactions

xhlucacommented, Oct 16, 2021

@banyous Feel free to contribute a section in the user guide about using language detection, and from there, if we feel a wrapper around fasttext would make life easier, then I’m happy to welcome a PR to add language detection to dlt.utils or dlt.lang

I think this is a decent starting point: https://fasttext.cc/docs/en/language-identification.html