Pytorch Vs Onnx: Pytorch is faster and provides different output

Environment info

• transformers version: 3.5.0
• Platform: Linux-4.19.112±x86_64-with-Ubuntu-18.04-bionic
• Python version: 3.6.9
• PyTorch version (GPU?): 1.7.0+cu101 (True)
• Tensorflow version (GPU?): 2.3.0 (True)
• Using GPU in script?: Yes
• Using distributed or parallel set-up in script?: No

Who can help

albert, bert, GPT2, XLM: @LysandreJik Onnx: @mfuntowicz

Information

Model I am using (Bert, XLNet …):

The problem arises when using:

• the official example scripts: (give details below)
• my own modified scripts: (give details below)

The tasks I am working on is:

• an official GLUE/SQUaD task: (give the name)
• my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

https://colab.research.google.com/drive/1UwgWgUF4k_GPJ5TcziHo4eH_rRFQeNVL?usp=sharing

Expected behavior

I have followed the the onnx export tutorial: https://github.com/huggingface/transformers/blob/master/notebooks/04-onnx-export.ipynb However, I have found 2 issues:

1. Pytorch is faster than Onnx.
2. Onnx produce different embedding output than Pytorch.

Could anyone help me to figure out the issue ?

[agemagician]

I have figured out the problem, but I don’t have the solution. When you use a single sample per batch it works correctly, but when you use more than one sample per batch, the results are totally different.

[github-actions[bot]]

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.