Big Bird generate() "local variable 'next_tokens' referenced before assignment"

I am facing this problem when doing text summarization. I am using google/bigbird-roberta-base and I get the following error when calling model.generate(input, max_length = 4096, num_beams=4, early_stopping=True, length_penalty = 0.8):

Input length of input_ids is 4096, but ``max_length`` is set to 4096.This can lead to unexpected behavior. You should consider increasing ``config.max_length`` or ``max_length``.
---------------------------------------------------------------------------
UnboundLocalError                         Traceback (most recent call last)
<ipython-input-13-90a633800ba7> in <module>()
----> 1 get_ipython().run_cell_magic('time', '', ' \ni = 0\nsize = 1\nout = []\nend = False\nprint_iters = 100\nsave_iters = 5\n \nwhile True:\n  if (i+size) >= n:\n    last = n\n    end = True\n  else:\n    last = i + size   \n \n  result = make_gen(   model_sum, tokens[i:last, :].detach().clone()  )\n \n  for j in range(result.shape[0]):\n    out.append(result[j])\n \n  if last % (print_iters*size) == 0:\n    print(last)\n    gc.collect()\n    torch.cuda.empty_cache()\n    torch.cuda.synchronize()\n    if last % (print_iters*size*save_iters) == 0:\n      with open(path_output + name + ".pkl", \'wb\') as f:\n        pickle.dump(out, f)\n      print("Saved to disk")\n \n  if end:\n    break\n  i = last')

6 frames
<decorator-gen-53> in time(self, line, cell, local_ns)

<timed exec> in <module>()

/usr/local/lib/python3.7/dist-packages/transformers/generation_utils.py in beam_search(self, input_ids, beam_scorer, logits_processor, stopping_criteria, max_length, pad_token_id, eos_token_id, output_attentions, output_hidden_states, output_scores, return_dict_in_generate, **model_kwargs)
   1808 
   1809         sequence_outputs = beam_scorer.finalize(
-> 1810             input_ids, beam_scores, next_tokens, next_indices, pad_token_id=pad_token_id, eos_token_id=eos_token_id
   1811         )
   1812 

UnboundLocalError: local variable 'next_tokens' referenced before assignment

[OscarGarciaF]

@vasudevgupta7 I was using AutoModelForSeq2SeqLM (this is what you use for summarization right?)

I have now changed to EncoderDecoderModel but now I face a new error

      1 input = tokens[0:1, :].to(device)
----> 2 generated = model_sum.generate(input, decoder_start_token_id = model_sum.config.decoder.pad_token_id, max_length = 512, num_beams = 4, early_stopping = True)

10 frames
/usr/local/lib/python3.7/dist-packages/transformers/models/big_bird/modeling_big_bird.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds, past_key_values_length)
    293 
    294         position_embeddings = self.position_embeddings(position_ids)
--> 295         embeddings += position_embeddings
    296 
    297         embeddings = self.dropout(embeddings)

RuntimeError: output with shape [4, 1, 768] doesn't match the broadcast shape [4, 0, 768]

[ilos-vigil]

I also got exact same error (output with shape...) when i generate on custom BigBird model. I fixed it by reducing model_max_length value from 4096 to 4094 and afterwards i can use pipeline for inference without any problem.

>>> tokenizer.max_len_single_sentence
4094
>>> tokenizer.model_max_length
4096
>>> tokenizer.model_max_length = 4094