XLM tokenizer lang2id attribute is None

Environment info

• transformers version: 4.5.1
• Platform: Windows-10-10.0.19041-SP0
• Python version: 3.8.8
• PyTorch version (GPU?): 1.8.1+cpu (False)
• Tensorflow version (GPU?): not installed (NA)
• Using GPU in script?: <fill in>
• Using distributed or parallel set-up in script?: No

Who can help

Models:

• albert, bert, xlm: @LysandreJik

Library:

• tokenizers: @LysandreJik

Information

Model I am using XLM with Causal language modelling:

The problem arises when using:

• the official example scripts: (give details below)

To reproduce

Steps to reproduce the behaviour:

1. Run example code from https://huggingface.co/transformers/multilingual.html

import torch
from transformers import XLMTokenizer, XLMWithLMHeadModel

tokenizer = XLMTokenizer.from_pretrained("xlm-clm-enfr-1024")
model = XLMWithLMHeadModel.from_pretrained("xlm-clm-enfr-1024")

language_id = tokenizer.lang2id['en']

The attribute lang2id is None and so I get a Nonetype is a non-suscriptable error. Following the example I am expecting to get 0 for language_id.

As a side note, it says that these checkpoints require language embeddings which I’m assuming is from the argument langs. What is the default behavior when this is not provided? I tried looking at https://huggingface.co/transformers/glossary.html but could not find any reference to it.

[LysandreJik]

Hello! Sorry for taking so long to get back to this issue - the issue should normally be fixed now, for all versions. We updated the configurations of the XLM models on the hub.

Thanks for flagging!

[cbaziotis]

FYI, I tried downgrading and I found that the most recent version that doesn’t have this bug is transformers==4.3.3. So you could try downgrading to that version for now, until someone fixes it.

pip install transformers==4.3.3