```input_embeds``` keyword not working properly (GPT2)

transfomer-cli env command was not recognized so I do not know which environment I have.

• transformers version: latest
• Platform: windows 10
• Python version: 3.8.5
• PyTorch version (GPU?): CPU
• Tensorflow version (GPU?): None
• Using GPU in script?: No
• Using distributed or parallel set-up in script?: No

Who can help

@patrickvonplaten, @LysandreJik

Information

Model I am using (GPT2):

When I want to provide my own embeded representation instead of input indices to a GPT2 that I want to train from scratch, I get the following error:

Traceback (most recent call last):

  File "<ipython-input-18-d0df910b9d57>", line 10, in <module>
    model(inputs_embeds=inputs_embeds)  # runs without error

  File "C:\Users\cnelias\anaconda3\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)

TypeError: forward() got an unexpected keyword argument 'inputs_embeds'

It does not come from my implementation since the following code also throws the same error:

from transformers import GPT2Model, GPT2Tokenizer
import torch

model = GPT2Model.from_pretrained('gpt2')
tokenizer = GPT2Tokenizer.from_pretrained('gpt2', pad_token='<PAD>')

input_ids = tokenizer.encode("Hello, how are you?", return_tensors='pt')
inputs_embeds = model.wte(input_ids)

model(inputs_embeds=inputs_embeds) 

I installed pytorch via conda, this could be the cause.

[qqaatw]

I think you probably checked a wrong docs version, the GPT2 in v2.1.1 docs was still under construction at that moment.

You might misunderstand my point. Here I mentioned above was not the latest docs, it was the version v2.1.1 you were previously using. At that moment (v2.1.1), the docs was still under construction which didn’t indicate whether input_embeds is usable or not.

So I checked out the codebase, the conclusion is: The input_embeds argument is not supported in v2.1.1 but is supported in the latest version v4.x.

Hope this clarification is helpful.

[qqaatw]

I’m sorry to hear that. I can install all three packages on windows without any error, therefore, I’m pretty sure that torchaudio is supported.

Maybe this is an environment problem on your system, you can try directly downloading whl packages from the index https://download.pytorch.org/whl/torch_stable.html and then install them manually.