How to provide multilingual support

Text preprocessing might be very language-dependent.

It would be great for Texthero to offer text preprocessing in all different languages.

There are probably two ways to support multilanguage:

from texthero.lang import hero_lang

from texthero import hero
hero.config.set_lang("lang")

We might have also cases where the dataset is composed of many languages. What would be the best solution in that case?

The first question we probably have to solve is: does different languages requires very different preprocessing pipeline and therefore different functions?

[ryangawei]

I’ll start implementing texthero.lang.hero_cn to make all functions support Chinese. If #60 is completed I’ll refactor the code to accommodate this feature. Is that OK?

[cedricconol]

I think @AlfredWGA’s solution of having a global setting is a better idea than adding language parameter for every function as i suggested and is also more feasible.

#3 is also very interesting and might be an even better idea as it automates the process. It aligns perfectly well to Texthero’s purpose of being easy to use and understand.