RobertaTokenizerFast does not add special tokens

I’m not sure whether this should be a part of tokenizers or transformers, because it uses both. Classes that don’t work are from transformers so I’m posting it here.

Environment info

• transformers version: 4.3.3
• Platform: Colab
• PyTorch version (GPU?): n/a
• Tensorflow version (GPU?): n/a
• Using GPU in script?: no
• Using distributed or parallel set-up in script?: no

Who can help

• tokenizers: @n1t0, @LysandreJik

Information

Reproduction code

https://colab.research.google.com/drive/1iYLBLzXRkQpdPyVlIdi_qNCzfbD1uwGs?usp=sharing

When loading tokenizer trained using tokenizers from transformers, e.g.

tfast = RobertaTokenizerFast.from_pretrained("./workdir/tokenizer", model_max_length=10)

it does not add special tokens

tfast("asd", add_special_tokens=True)

{'input_ids': [400, 72], 'attention_mask': [1, 1]}

“Slow” version behaves correctly:

tslow = RobertaTokenizer.from_pretrained("./workdir/tokenizer", model_max_length=10)
tslow("asd", add_special_tokens=True)

{'input_ids': [0, 400, 72, 2], 'attention_mask': [1, 1, 1, 1]}

Expected behavior

Both tokenizers produce the same output.

[LysandreJik]

Yes, I believe that is so. Tokenizers created with tokenizers need to have their post-processors/pre-tokenizers and other components defined to work correctly, otherwise it yields unexpected results as we have just seen!

[marrrcin]

Closing, but still seems odd that the behaviour for exact same files is different between those tokenizers…