2.9.0 Padding Bug

🐛 Bug

Information

Model I am using (Bert, XLNet …): Roberta with RobertaForSequenceClassification head

Language I am using the model on (English, Chinese …): Polish

The problem arises when using:

• the official example scripts: (give details below)
• my own modified scripts: (give details below)

The problem arises when performing a forward pass while training. I assume something is wrong with padding, though with 2.8.0 version it works correctly.

The tasks I am working on is:

• an official GLUE/SQUaD task: (give the name)
• my own task or dataset: (give details below)

I’m using pre-trained roBERTa model for classification task. Details can be found in the notebook below.

To reproduce

Steps to reproduce the behavior:

1. Run https://colab.research.google.com/drive/1w54ScQ3-dh99-cO4OGG6-IdP4JRfki59?usp=sharing

Error message:

     56         # Perform a forward pass
---> 57         loss, logits = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)

...

   2033     """
   2034     # The series of casts and type-conversions here are carefully balanced to both work with ONNX export and XLA.
-> 2035     mask = input_ids.ne(padding_idx).int()
   2036     incremental_indices = torch.cumsum(mask, dim=1).type_as(mask) * mask
   2037     return incremental_indices.long() + padding_idx

TypeError: ne() received an invalid combination of arguments - got (NoneType), but expected one of:
 * (Tensor other)
      didn't match because some of the arguments have invalid types: (!NoneType!)
 * (Number other)
      didn't match because some of the arguments have invalid types: (!NoneType!)

Expected behavior

It should work without errors. The problem arises after installing 2.9.0 version. With 2.8.0 version it works correctly.

Environment info

• transformers version: 2.9.0
• Platform: Google colab
• Python version: 3.6.9
• PyTorch version (GPU?): 1.5.0+cu101
• Tensorflow version (GPU?): 2.2.0rc4
• Using GPU in script?: Yes
• Using distributed or parallel set-up in script?: n/a

[dsindex]

trouble shooting

position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx).to(input_ids.device)
mask = input_ids.ne(padding_idx).int()
...
ne() received an invalid combination of arguments - got (NoneType)

just modify ‘pad_token_id: null’ to ‘pad_token_id: 1’ in config.json

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.