Different vocab_size between model and tokenizer of mT5

Environment info

• transformers version: 4.1.1
• Platform: ubuntu 18.04
• Python version: 3.8.5
• PyTorch version (GPU?): 1.7.1

Who can help

@patrickvonplaten

To reproduce

Steps to reproduce the behavior:

from transformers import AutoModelForSeq2SeqLM
from transformers import AutoTokenizer

mt5s = ['google/mt5-base', 'google/mt5-small', 'google/mt5-large', 'google/mt5-xl', 'google/mt5-xxl']

for mt5 in mt5s:
    model = AutoModelForSeq2SeqLM.from_pretrained(mt5)
    tokenizer = AutoTokenizer.from_pretrained(mt5)

    print()
    print(mt5)
    print(f"tokenizer vocab: {tokenizer.vocab_size}, model vocab: {model.config.vocab_size}")

This is problematic in case when one addes some (special) tokens to tokenizer and resizes the token embedding of the model with model.resize_token_embedding(len(tokenizer))

Expected behavior

vocab_size for model and tokenizer should be the same?

[LysandreJik]

Hello! This is a duplicate of https://github.com/huggingface/transformers/issues/4875, https://github.com/huggingface/transformers/issues/10144 and https://github.com/huggingface/transformers/issues/9247

@patrickvonplaten, maybe we could do something about this in the docs? In the docs we recommend doing this:

model.resize_token_embedding(len(tokenizer))

but this is unfortunately false for T5!

[takiholadi]

Hello! This is a duplicate of #4875, #10144 and #9247

@patrickvonplaten, maybe we could do something about this in the docs? In the docs we recommend doing this:

model.resize_token_embedding(len(tokenizer))

but this is unfortunately false for T5!

What is the correct way to resize_token_embedding for T5/mT5?