fast tokenizer issue on most user uploaded models

Environment info

• transformers version: 3.4.0
• Platform: Linux-5.8.0-25-generic-x86_64-with-glibc2.32
• Python version: 3.8.6
• PyTorch version (GPU?): 1.6.0 (True)
• Tensorflow version (GPU?): not installed (NA)
• Using GPU in script?: yes
• Using distributed or parallel set-up in script?: yes

Who can help

@mfuntowicz @julien-c

Information

Found the bug on camembert/camembert-base-ccnet but probably common to many models uploaded by users. On camembert base model, it works out of the box (there is no bug).

To reproduce

Since tokenizer 0.9, it’s possible to load the many unigram based tokenizers with the fast Rust implementation.
It appears that the file tokenizer_config.json of some of them is not up to date, in particular the information "model_max_length": 512 is missing.

Because of that, the value of model_max_length is a very big integer.

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("camembert/camembert-base-ccnet", use_fast=True)
tokenizer.model_max_length
# Out[4]: 1000000000000000019884624838656

To fix it, the field model_max_length has to be added to the config file.

Expected behavior

I would expect tokenizer.model_max_length to be equal to 512.

[julien-c]

Yes, we need to remove all the hardcoded configuration values of tokenizers in the transformers source code, and upload tokenizer_config.json files for all those models.

Also cc @n1t0

[Narsil]

@pommedeterresautee

Hi, I am not sure it’s a fast tokenizers bug but maybe more a property that was (maybe unintentionnally) dropped from Tokenizers.

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("camembert/camembert-base-ccnet", use_fast=False)
tokenizer.model_max_length
# Out[4]: 1000000000000000019884624838656

Can you tell us what’s the actual bug for you in the end ? Just to make sure the fix I am working on will actually work as generally as possible