past_key_values tuple index out of range error when using text2text-generation pipeline with encoder-decoder model

Environment info

• transformers version: 4.3.0
• Platform: Linux-5.4.0-65-generic-x86_64-with-Ubuntu-20.04-focal
• Python version: 3.7.9
• PyTorch version (GPU?): 1.7.0 (False)
• Tensorflow version (GPU?): not installed (NA)
• Using GPU in script?: No
• Using distributed or parallel set-up in script?: No

Who can help

@LysandreJik

Information

Model I am using (Bert, XLNet …): I am using the encoder-decoder model with a Roberta encoder and RobertaForCausalLM decoder.

The problem arises when using:

• the official example scripts: (give details below)
• my own modified scripts: (give details below)

In my real code I am using custom pre-trained models and tokenizers, but the error and behavior is the same as that produced by the demo script below.

The tasks I am working on is:

• an official GLUE/SQUaD task: (give the name)
• my own task or dataset: (give details below)

I am trying to use a pipeline to generate results from an encoder-decoder model that was trained on a custom text2text dataset.

To reproduce

Steps to reproduce the behavior:

You can just run the script below, or:

1. Load an encoder-decoder model with RoBERTa encoder and decoder
2. Create a text2text-generation pipeline with an appropriate tokenizer
3. Use the pipeline to generate a result

import transformers

encoder = transformers.RobertaModel.from_pretrained(pretrained_model_name_or_path='roberta-base')
decoder = transformers.RobertaForCausalLM.from_pretrained(pretrained_model_name_or_path='roberta-base')

encoder_decoder_model = transformers.EncoderDecoderModel(encoder=encoder, decoder=decoder)

tokenizer = transformers.AutoTokenizer.from_pretrained('google/roberta2roberta_L-24_bbc')

text2text = transformers.pipeline('text2text-generation', model=encoder_decoder_model, tokenizer=tokenizer)

output = text2text('This is a test sentence.')

print(output)

Output:

If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.`
normalizer.cc(51) LOG(INFO) precompiled_charsmap is empty. use identity normalization.
Traceback (most recent call last):
  File "demo.py", line 12, in <module>
    output = text2text('This is a test sentence.')
  File "/home/james/Code/demo/.env/lib/python3.7/site-packages/transformers/pipelines/text2text_generation.py", line 125, in __call__
    **generate_kwargs,
  File "/home/james/Code/demo/.env/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
    return func(*args, **kwargs)
  File "/home/james/Code/demo/.env/lib/python3.7/site-packages/transformers/generation_utils.py", line 913, in generate
    **model_kwargs,
  File "/home/james/Code/demo/.env/lib/python3.7/site-packages/transformers/generation_utils.py", line 1177, in greedy_search
    output_hidden_states=output_hidden_states,
  File "/home/james/Code/demo/.env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/james/Code/demo/.env/lib/python3.7/site-packages/transformers/models/encoder_decoder/modeling_encoder_decoder.py", line 430, in forward
    **kwargs_decoder,
  File "/home/james/Code/demo/.env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/james/Code/demo/.env/lib/python3.7/site-packages/transformers/models/roberta/modeling_roberta.py", line 937, in forward
    return_dict=return_dict,
  File "/home/james/Code/demo/.env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/james/Code/demo/.env/lib/python3.7/site-packages/transformers/models/roberta/modeling_roberta.py", line 771, in forward
    past_key_values_length = past_key_values[0][0].shape[2] if past_key_values is not None else 0
IndexError: tuple index out of range

Expected behavior

I expect the pipeline to generate an output string.

[sailist]

The same exception may also be raised when model is in train mode, call model.eval() before may solve this problem. It happened when I use model BartForConditionalGeneration.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.