BertTokenizer with BertJapaneseTokenizer pretrained model generates unintended tokenization.

Environment info

• transformers version: 4.9.0.dev0
• Platform: Windows-10-10.0.19043-SP0
• Python version: 3.9.5
• PyTorch version (GPU?): 1.9.0+cpu (False)
• Tensorflow version (GPU?): 2.5.0 (False)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: <fill in>
• Using distributed or parallel set-up in script?: <fill in>

Who can help

Information

BertTokenizer with BertJapaneseTokenizer pretrained model generates unintended tokenization without any caution.

To reproduce

Steps to reproduce the behavior:

Run

EXAMPLE_BERT_JAPANESE_ID = "cl-tohoku/bert-base-japanese"
tokenizer = BertTokenizer.from_pretrained(EXAMPLE_BERT_JAPANESE_ID)
print(tokenizer.tokenize("今日はいい天気ですね"))

Expected behavior

not_correct = BertTokenizer.from_pretrained(EXAMPLE_BERT_JAPANESE_ID)
correct = BertJapaneseTokenizer.from_pretrained(EXAMPLE_BERT_JAPANESE_ID)
print(not_correct.tokenize("今日はいい天気ですね"))
print(correct.tokenize("今日はいい天気ですね"))

Because the two tokenizers were made from the same pretrained model, the output should have been

['今日', 'は', 'いい', '天気', 'です', 'ね']
['今日', 'は', 'いい', '天気', 'です', 'ね']

or BertTokenizer.from_pretrained(EXAMPLE_BERT_JAPANESE_ID) should have raised an error. However, the actual result was

['今', '日', 'はい', '##い', '天', '気', 'です', '##ね']
['今日', 'は', 'いい', '天気', 'です', 'ね']

and no error or warning raised.

[SaulLu]

Thank you very much for offering to take care of this issue!

From my point of view, what you described above sounds really great! 👍

[europeanplaice]

@SaulLu Thank you for your offer. I want to try to tackle this problem. I plan to add something like below https://github.com/huggingface/transformers/blob/122d7dc34fd0e397a08b8a584a632fc57d3fd5d0/src/transformers/models/auto/tokenization_auto.py#L527-L551

to from_pretrained in PreTrainedTokenizerBase (tokenization_utils_base.py) to make sure that we can check whether a user is trying to use different tokenizers between cls and config.json or tokenizer_config.json’s class before a tokenizer returns. If this detected conflicts between them, a warning would be logged, or an error would occur.

I want my PR to be in line with your overall plan, so I hope to get your opinion about this comment.