UnboundLocalError: local variable 'next_tokens' referenced before assignment when using Generate()

🐛 Bug

I have pre-trained GPT2 on a summarisation dataset such that summarisation is a language modelling task i.e. input = concat( padded_to_max_len(body , “TL;DR:” , summary)).

For some reason, this error occurs when I try to generate via beam search using GPT2’s with language modelling head. Here is my code:

from transformers import AutoTokenizer, GPT2LMHeadModel
import torch

# define tokenizer
tokenizer_kwargs = {"bos_token": "<|startoftext|>", "eos_token": "<|endoftext|>", "pad_token": "<|pad|>"}
tokenizer = AutoTokenizer.from_pretrained("gpt2", **tokenizer_kwargs)

# define model
model = GPT2LMHeadModel.from_pretrained("gpt2", pad_token_id=tokenizer.eos_token_id)

# define input 
input_ids = torch.tensor(tokenizer.encode("some text that ends in TL;DR:")).unsqueeze(0)

# attempt to generate
y_pred_tensor = model.generate(input_ids=input_ids,
                                                num_beams=5,
                                                early_stopping=True,
                                                no_repeat_ngram_size=2,
                                                max_length=100
                                                )

File "/Users/christopherdoyle/cp_projects/scribbl-ai/Scribbl/Scribbl/summarizer/models.py", line 147, in summarize
    y_pred_tensor = self.model.generate(input_ids=input_tensor,
  File "/Users/christopherdoyle/cp_projects/scribbl-ai/Scribbl/venv/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 15, in decorate_context
    return func(*args, **kwargs)
  File "/Users/christopherdoyle/cp_projects/scribbl-ai/Scribbl/venv/lib/python3.8/site-packages/transformers/modeling_utils.py", line 1100, in generate
    output = self._generate_beam_search(
  File "/Users/christopherdoyle/cp_projects/scribbl-ai/Scribbl/venv/lib/python3.8/site-packages/transformers/modeling_utils.py", line 1499, in _generate_beam_search
    (token_id % vocab_size).item() is not eos_token_id for token_id in next_tokens[batch_idx]
UnboundLocalError: local variable 'next_tokens' referenced before assignment

Model I am using: GPT2LMHeadModel

Language I am using the model on (English, Chinese …): en

Rather strangely, it works when max_length = 1024, but not with smaller values.

To reproduce

Running on CPU python==3.8, transformers==2.11.0, torch==1.5.0

[patrickvonplaten]

The problem is that max_length is not bigger than cur_len so that model will not produce any text. This will fix the problem:

outputs = model.generate(input_ids=input_ids, num_beams=3, max_length=75)

[chrisdoyleIE]

Hey @patrickvonplaten ,

I’ll do some digging and see if I can’t reproduce it myself such that it’s easily paste-able and then I can share this code (I currently have a few custom packages calling eachother which is hairier than i’d like and not trivial to insert into an issue).