Remove Diacritics for Urdu Language

I would like to contribute for Urdu language support, let me start with simple issue now,

For the Urdu text with diacritics text = “اِس, اُس”

following code produces incorrect output

import pandas as pd 
import re
import texthero as hero
text = "اِس, اُس"
s = pd.Series(text)
s1 = hero.remove_diacritics(s)
s1
is, us

produces the output is, us which is not the intended., but it is transliterated output.

The intented output is اس, اس

Probably which can be acheived by replacing following diacritics char

Urdu Diacritics zabar = u’\u064e’ pesh = u’\u064f’ zer = u’\u0650’ tashdid = u’\u0651’ jazam = u’\u0652’

[henrifroese]

I believe the issue is that the proposed changes are on the Fix-Remove-Diacritics branch of the repository. With your command, pip uses the master branch where the changes are not yet implemented. You can try using !pip install git+https://github.com/SummerOfCode-NoHate/texthero.git@Fix_Remove_Diacritics, I think that should work. If not, you could just copy the functions I pasted in #72 and try them out directly.

[cmhashim]

from texthero.ur import hero
hero.remove_diacritics(...)

Where this remove_diacritics is specialized in dealing with Urdu text.

I think its time to do this. It was my mistake i gave very simple example of Urdu text with diacritics, but it much more complex to handle diacritics in Urdu/Arabic. Some diacritics are part of Urdu words, and it must be written, and some can be excluded. Hence, can we have a optional argument, to exclude/include a list of diacritics to retain/remove it.

Some Examples:

retain_diacritics_eg_text = “فوراً, حتیٰ, آزاد, ہوئی”