LEDForSequenceClassification fine-tuning model gives: IndexError: index out of range in self

System Info

transformers - 4.21.1 Python - 3.8.13 torch - 1.12.0+cu113

Who can help?

@patrickvonplaten

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, …)
• My own task or dataset (give details below)

Reproduction

I’m trying to fine-tune the LED model for the SequenceClassification task.

It works when I use max_sequence_length = 1024 but it doesn’t when it goes beyond these numbers. D

After debugging, I found a similar issue #14312, and tried applying the proposed solution by adding decoder_input_ids even though it doesn’t make sense to me that we really need this as an input.

To reproduce:

from transformers import LEDTokenizer, LEDForSequenceClassification
  
tokenizer = LEDTokenizer.from_pretrained("allenai/led-base-16384")
model = LEDForSequenceClassification.from_pretrained("allenai/led-base-16384")

# this works (tokens < 1024)
inputs = tokenizer("HuggingFace"*1000, return_tensors="pt")
with torch.no_grad():
      model(**input)

# this does not work! (tokens > 1024)
inputs = tokenizer("HuggingFace"*2048, return_tensors="pt")
with torch.no_grad():
      model(**input)

# this does not work as well! (tokens > 1024)
inputs = tokenizer("HuggingFace"*2048, return_tensors="pt")
inputs['decoder_input_ids'] = inputs['input_ids'][:512]
with torch.no_grad():
      model(**input)

Without adding decoder_input_ids to tokenized inputs, It simply complains of Index out of range while calling torch.embedding.

with decoder_input_ids as suggested in the reference issue above, It throws an IndexError: The Shape of the mask [1, 1026] at index 1 does not match the shape of the indexed tensor [1, 512, 768] at index 1

Expected behavior

As LEDForSequenceClassification should work for a sequence length of more than 1024 tokens.

[patrickvonplaten]

Hey @darshan2203,

Sorry I won’t be free anytime soon to look into this issue - @ArthurZucker do you want to give it a try?

[ArthurZucker]

Oh right! Sorry will have a look asap 🤗