[docs]Incorrect way of input encoding for "multiple choice" models in documentation?

In the documentation about “xxForMultipleChoice” models like BERT, ALBERT, RoBERTa, the examples goes like this:

>>> from transformers import BertTokenizer, BertForMultipleChoice
>>> import torch

>>> tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
>>> model = BertForMultipleChoice.from_pretrained('bert-base-uncased')

>>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
>>> choice0 = "It is eaten with a fork and a knife."
>>> choice1 = "It is eaten while held in the hand."
>>> labels = torch.tensor(0).unsqueeze(0)  # choice0 is correct (according to Wikipedia ;)), batch size 1

>>> encoding = tokenizer([[prompt, prompt], [choice0, choice1]], return_tensors='pt', padding=True)
>>> outputs = model(**{k: v.unsqueeze(0) for k,v in encoding.items()}, labels=labels)  # batch size is 1
...

In current version (4.5.1), the encoding actually consists of 2 sentences: [prompt + prompt], and [choice0 + choice1], which to my knowledge is incorrect, as each encoded sentence should include one prompt and one choice. I think the encoding supposed to be like:

tokenizer([[prompt, choice0], [prompt, choice1]], return_tensors='pt', padding=True)

So, is there anything wrong?

[sgugger]

This has been fixed, but you need to switch to the master documentation to see the change.

[mbforbes]

I’m running into this issue as well.

I’m not super familiar working with multiple choice models, but I think that given #6074, and the run_swag.py example, these should be passed as two lists, instead of one.

In other words, in the example, instead of

encoding = tokenizer([[prompt, prompt], [choice0, choice1]], return_tensors='pt', padding=True)

it should be

encoding = tokenizer([prompt, prompt], [choice0, choice1], return_tensors='pt', padding=True)

It would be awesome to make this two-character-deletion change, as it just tripped me up when starting working on a multiple choice model!