Tokenization of punctuation in Hebrew and other non-latin languages

When tokenizing Hebrew, the full stop at the end of a sentence is not tokenized, while if the sentence ends with either a question mark, an exclamation mark, or ellipses, those marks are tokenized.

Example:

from spacy.he import Hebrew

tokenizer = Hebrew().tokenizer

print(list(w.text for w in tokenizer('עקבת אחריו בכל רחבי המדינה.')))
#  ['עקבת', 'אחריו', 'בכל', 'רחבי', 'המדינה.']

print(list(w.text for w in tokenizer('עקבת אחריו בכל רחבי המדינה?')))
#  ['עקבת', 'אחריו', 'בכל', 'רחבי', 'המדינה', '?']

print(list(w.text for w in tokenizer('עקבת אחריו בכל רחבי המדינה!')))
#  ['עקבת', 'אחריו', 'בכל', 'רחבי', 'המדינה', '!']

print(list(w.text for w in tokenizer('עקבת אחריו בכל רחבי המדינה..')))
#  ['עקבת', 'אחריו', 'בכל', 'רחבי', 'המדינה', '..']

print(list(w.text for w in tokenizer('עקבת אחריו בכל רחבי המדינה...')))
#  ['עקבת', 'אחריו', 'בכל', 'רחבי', 'המדינה', '...']

Info about spaCy

• spaCy version: 1.8.0
• Platform: Linux-3.13.0-74-generic-x86_64-with-debian-jessie-sid
• Python version: 3.6.1
• Installed models:

[beneyal]

I’ll take a shot at fixing it.

[ines]

Thanks for the report. I think this is caused by the global regex rules for punctuation, some of which currently only cover latin characters. We originally chose the approach of spelling out the individual characters because it made it easier to create uppercase/lowercase sets, and kept things a bit more readable while we were tidying up the language data and inviting more people to contribute.

But now that we’re adding more and more languages, this keeps coming up so we should fix this. (If I remember correctly, this was already causing problems for people working with Bengali and developing Russian integration.)

~I’ll open a separate issue about this for spaCy v2.0, but in short:~ Never mind, just making this the master issue. Steps are:

• get rid of explicit character list
• use regex library to handle compiling the correct character classes