Indian Tokenizer should recognise deergha virama ("॥") as a SINGLE token.

There are two special punctuation marks in Indian Languages, namely the the purna virama (“|”) and deergha virama (“॥”), for Indian language scripts. While indian_tokenizer.py does a good job in tokenizing, it should recognise the deergha virama (“॥”) as a single token instead of two tokens.

>>> from cltk.tokenize.indian_tokenizer import indian_punctuation_tokenize_regex as i_word
#Sanskrit
>>> sentence = "अग्निमीळे पुरोहितं यज्ञस्य देवं रत्वीजम |होतारं रत्नधातमम ||"
>>> sanskrit_text_tokenize = i_word(sentence)
>>> sanskrit_text_tokenize 
['अग्निमीळे', 'पुरोहितं', 'यज्ञस्य', 'देवं', 'रत्वीजम', '|', 'होतारं', 'रत्नधातमम', '|', '|']
#Bengali
>>> sentence = "রাজপণ্ডিত হব মনে আশা করে | সপ্তশ্লোক ভেটিলাম রাজা গৌড়েশ্বরে ||">>> bengali_text_tokenize = i_word(sentence)>>> bengali_text_tokenize 
['রাজপণ্ডিত', 'হব', 'মনে', 'আশা', 'করে', '|', 'সপ্তশ্লোক', 'ভেটিলাম', 'রাজা', 'গৌড়েশ্বরে', '|', '|']

[lazycoder1]

In Sanskrit especially in poems, paragraphs end with || . I dont know about other languages though.

[nikheelpandey]

Well at least in Hindi, the sentence usually ends up with’|'. I mean I have always seen the sentences ended with a ‘|’ but have never seen something ending with ‘||’ unless it is a peculiar form of poem writing.