Spacy fails to tokenize the closing parenthesis as a suffix if preceded by 8: "8)"

How to reproduce the behaviour

import spacy; 
nlp = spacy.load('en_core_web_sm')
doc = nlp("(8)")
[tok.text for tok in doc]

returns: ['(', '8)']

Your Environment

Ubuntu 18.04.4 LTS

• Operating System: Linux
• Python Version Used: 3.6.9
• spaCy Version Used: Spacy 2.2.3 (but also happens with Spacy 2.3.0)
• Environment Information:

[adrianeboyd]

The emoticon list could definitely be shortened.

Two tips:

1. To see why the tokenizer is tokenizing a particular way, use nlp.tokenizer.explain:

print(nlp.tokenizer.explain("(8)"))
# [('PREFIX', '('), ('SPECIAL-1', '8)')]

1. To remove a special case (which are called rules internally):

nlp.tokenizer.rules = {k: v for k, v in nlp.tokenizer.rules.items() if k != "8)"}
print(nlp.tokenizer.explain("(8)"))
# [('PREFIX', '('), ('TOKEN', '8'), ('SUFFIX', ')')]

(I’m recommending doing it this way with a dict comprehension since reassigning the rules property also clears the internal tokenizer cache, which del nlp.tokenizer.rules["8)"] wouldn’t. This is not the best state of affairs, but it’s how it works for now.)

[adrianeboyd]

Yes, you can use del like normal if you do it before initializing the tokenizer. It’s mainly if you’re editing the settings for a tokenizer that’s already been initialized. If you haven’t tokenized anything yet it’s also fine, but if you’ve already used the model to tokenize a few examples and are modifying it on-the-fly, you can run into some really confusing cache-related behavior.