Transformer-XL tokenizer cannot properly tokenize brackets

🐛 Bug

Information

The TransfoXLTokenizer is not able to tokenize words with surrounding brackets correctly. I compared it with the BertTokenizer from bert-base-uncased which gives the expected result. Example text is: "Hello (bracket)"

Model I am using: Transformer-XL

Language I am using the model on: English

The problem arises when using:

• my own modified scripts

To reproduce

Steps to reproduce the behavior:

from transformers import BertTokenizer, TransfoXLTokenizer

bert = BertTokenizer.from_pretrained('bert-base-uncased')
transfoxl = TransfoXLTokenizer.from_pretrained('transfo-xl-wt103')

def test_bracket(tokenizer):
    enc = tokenizer.encode("Hello (bracket)")
    dec = tokenizer.decode(enc)
    print(f"ORG: Hello (bracket)\nENC: {enc}\nDEC: {dec}")

Results:
test_bracket(bert) gives the following output:

ORG: Hello (bracket)
ENC: [101, 7592, 1006, 21605, 1007, 102]
DEC: [CLS] hello ( bracket ) [SEP]

test_bracket(transfoxl) gives the following output:

ORG: Hello (bracket)
ENC: [14049, 24]
DEC: Hello <unk>

If the parameter add_space_before_punct_symbol=True is passed, then the result is:

ORG: Hello (bracket)
ENC: [14049, 24, 21]
DEC: Hello <unk> )

Expected behavior

The TransfoXLTokenizer should detect the punctuation symbols, e.g. (, separately and thus give the same result as the BertTokenizer (except the special tokens of course): hello ( bracket )

Environment info

• transformers version: 2.11.0
• Platform: Windows-10-10.0.18362-SP0
• Python version: 3.6.10
• PyTorch version (GPU?): 1.4.0 (False)
• Tensorflow version (GPU?): 2.1.0 (False)

[thomwolf]

Yes of course, so the new API didn’t touch any model-specific behavior, it was all about the user-facing up-stream methods.

In your case, I think you’ll probably want to update the _tokenize() method of Transfo-XL tokenizer here: https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_transfo_xl.py#L339-L356

This is the method in charge of splitting words in token strings.

You can have a look at the XLM tokenizer if you want to see how people have been using sacremoses: https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_xlm.py

[RafaelWO]

Thanks 😃