Tokenization_utils doesn't work with Pytorch-Lightning on 2.10.0 version

🐛 Bug

Information

Model I am using (Bert, XLNet …): Bert with pytorch-lightning

Language I am using the model on (English, Chinese …): English

The problem arises when using

• my own modified scripts: Pytorch Dataset with tokenizer inside

The tasks I am working on is:

• my own task or dataset

To reproduce

Take a look at this colab link.

I’ve copied the method from pytorch-lightning which shows an error on 2.10.0 transformers when process a batch. Doing the same with transformers 2.8.0 cause no error.

/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils.py in __getattr__(self, item)
    201 
    202     def __getattr__(self, item: str):
--> 203         return self.data[item]
    204 
    205     def keys(self):

KeyError: 'cuda'

Expected behavior

No error

• transformers version: 2.10.0
• Platform: Linux
• Python version: 3.6.9
• PyTorch version (GPU?): 1.5.0+cu101
• Using GPU in script?: Yes
• Using distributed or parallel set-up in script?: No

[mfuntowicz]

BatchEncoding is indeed a UserDict, if you want to access the actual dict, you can use the data attribute:

be = tokenizer.batch_encode_plus(...)
be.data

[sirily]

Thank you! So it’s not a bug, but an expected behaviour