device-side assert triggered Error while doing inference on Distilbert and Bert

Environment info

• transformers version: 3.4.0
• Platform: Colab
• Python version: 3.8
• PyTorch version (GPU?):
• Tensorflow version (GPU?):
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Who can help

Information

Model I am using Distilbert

The problem arises when using:

• the official example scripts: (give details below)
• my own modified scripts: (give details below) colab

The tasks I am working on is:

• an official GLUE/SQUaD task: SQUaD v2
• my own task or dataset:

To reproduce

Steps to reproduce the behavior:

1. Get Model and Tokenizer
2. Get SQUAD2 datasets
3. Performance Inference on the validation dataset with GPU
4. get result on SQUAD V2 Metrix

Run the colab to reproduce.

Expected behavior

It should give results in the below format without error on SQUAD V2 Metrix

{'exact': 79.4660153288975, 'f1': 82.91266052065696, 'total': 11873, 'HasAns_exact': 77.64844804318489, 'HasAns_f1': 84.55162253066118, 'HasAns_total': 5928, 'NoAns_exact': 81.27838519764508, 'NoAns_f1': 81.27838519764508, 'NoAns_total': 5945, 'best_exact': 79.4660153288975, 'best_exact_thresh': 1.0, 'best_f1': 82.91266052065693, 'best_f1_thresh': 1.0}

Note: This code is working fine for longformer model. I found this issue in Distilbert and Bert Model while doing inference on GPU

Tagging SMEs: @LysandreJik

[LysandreJik]

Glad we could help, closing!

[LysandreJik]

Hi @bhadreshpsavani, this can’t work on CPU. You’re sending a sequence that is too long to the model so it cannot handle it.

Please replace

inputs = tokenizer(example['question'], example['context'], return_tensors="pt")

by

inputs = tokenizer(example['question'], example['context'], return_tensors="pt", truncation=True)

This truncates the sequences that are too long.

Your colab should work then.