past functionality broken in release 2.9.0 and 2.9.1

🐛 Bug

Information

Model: gpt2

To reproduce

Steps to reproduce the behavior:

from transformers.tokenization_gpt2 import GPT2Tokenizer
from transformers.modeling_gpt2 import GPT2LMHeadModel
import torch

# Remember to run transformers with latest master (not release 2.5.1)
tokenizer = GPT2Tokenizer.from_pretrained('gpt2', pad_token='<|endoftext|>')
model = GPT2LMHeadModel.from_pretrained('gpt2')

# Complete phrases are: "I like to drink soda without sugar" and "Go watch TV alone, I am not going"
doc = "I like to"
# note: comment the above line and uncomment the following line to make it work with 1 document
docs_tensors = tokenizer.batch_encode_plus([doc], pad_to_max_length=True, return_tensors='pt')

docs_next = [" soda and ", " with this"]
# note: comment the above line and uncomment the following line to make it work with 1 document
docs_next_tensors = tokenizer.batch_encode_plus(
    [d for d in docs_next], pad_to_max_length=True, return_tensors='pt')

# predicting the first part of each phrase
_, past = model(docs_tensors['input_ids'], attention_mask=docs_tensors['attention_mask'])

# manipulating the past
past_expanded = [torch.repeat_interleave(layer, torch.LongTensor([2]), dim=1) for layer in past]
past_attention_mask = torch.ones(docs_next_tensors['attention_mask'].shape[0], len(docs_tensors['input_ids'][0]), dtype=torch.int64)
attn_mask = torch.cat([past_attention_mask, docs_next_tensors['attention_mask']], dim=-1)

# predicting the rest of the phrase with past
logits, _ = model(docs_next_tensors['input_ids'], attention_mask=attn_mask, past=past_expanded)
logits = logits[:, -1]
_, top_indices_results = logits.topk(50)

words = [tokenizer.decode([idx.item()]) for tir in top_indices_results for idx in tir]

print("Predictions for:", [doc + n for n in docs_next])
print("Results with past:", words)

#####################
docs_full_tensors = tokenizer.batch_encode_plus(
    [doc + n for n in docs_next], pad_to_max_length=True, return_tensors='pt')
logits, _ = model(docs_full_tensors['input_ids'], attention_mask=docs_full_tensors['attention_mask'])
logits = logits[:, -1]
_, top_indices_results = logits.topk(50)

words = [tokenizer.decode([idx.item()]) for tir in top_indices_results for idx in tir]

print("Predictions for:", [doc + n for n in docs_next])
print("Results without past:", words)

Expected behavior

I am expecting the past functionality to work. But I’m getting the following error:

RuntimeError: The size of tensor a (4) must match the size of tensor b (5) at non-singleton dimension 3

This was working correctly up to release 2.8.0

Environment info

• transformers version:
• Platform: Macos
• Python version: 1.4.0 / 1.5.0
• PyTorch version (GPU?): 2.9.0 cpu / 2.9.1 cpu
• Using GPU in script?: no
• Using distributed or parallel set-up in script?: no

[patrickvonplaten]

Hi @Damiox,

Sorry to answer so late. We had some discussion internally about it. And you are correct, we should revert the PR I merged earlier. This is done in #4581 and should be included in the next version. @thomwolf @LysandreJik

[Damiox]

I’m really looking forward to knowing if this can be fixed so to continue working as it was working up to v2.8.0. Any new thoughts on this? Thanks