discrepancy between the Huggingface T5Tokenizer and the original T5tokenizer

Environment info

• Transformers version: 4.3.2
• Platform: Linux
• Python version: 3.7
• PyTorch version (GPU?): yes
• Tensorflow version (GPU?): -
• Using GPU in script?: no
• Using distributed or parallel set-up in script?: no

Who can help

• tokenizers: @n1t0, @LysandreJik

Information

Model I am using T5Tokenizer, and I adapted the code of run_mlm.py [1] to use it with T5 tokenizer, when I run the code I am getting

This tokenizer does not have a mask token which is necessary for masked language modeling. "
ValueError: This tokenizer does not have a mask token which is necessary for masked language modeling. You should pass `mlm=False` to train on causal language modeling instead.

I checked the error and this is because: tokenizer.mask_token is None for T5Tokenizer, checking T5 paper, they use a masked language modeling with their seq2seq objective as the pretraining objective, so they must have trained a masked token as their paper says, could you give me some insight why masked token does not exist in huggingface implementation of T5Tokenizer and how I can correct this to be able to run run_mlm codes ? thank you

[1] https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py

To reproduce

tokenizer = AutoTokenizer.from_pretrained("t5-small")
print(tokenizer.mask_token) => this is None

Expected behavior

The masked token as per T5 paper should exist in T5Tokenizer.

[NielsRogge]

Hi,

T5 is an encoder-decoder Transformer. The run_mlm.py script can only be used for encoder-only models, such as BERT, RoBERTa, DeBERTa, etc.

Besides this, T5 does not use the regular [MASK] token as BERT. Rather than masked language modeling, T5 is pre-trained on “unsupervised denoising training”. This is explained here.

[lancekung]

Hi @NielsRogge thanks, but I have checked the paper of T5 and this seems to be a unique token:

“We consider two strategies to achieve this: First, instead of replacing each corrupted token with a mask token, we replace the entirety of each consecutive span of corrupted tokens with a unique mask token.”

As for using run_mlm.py script, I do not think T5 model can be an issue as if we could add T5ForConditionalGeneration it could work to me in run_mlm.py out of the box.

Is there any place I could look see how to create datasets the way you mentioned to do T5 pretraining with huggingface codes? thanks

HI, Did you successfully run the Huggingface T5 pretraining? Can you give me some advice?