convert_graph_to_onnx.convert broken for translation model facebook/wmt19-en-de

Environment info

• transformers version: 4.2.2
• Platform: Linux-4.15.0-132-generic-x86_64-with-glibc2.10
• Python version: 3.8.5
• PyTorch version (GPU?): 1.7.1 (True)
• Tensorflow version (GPU?): 2.5.0 (True)
• Using GPU in script?: True
• Using distributed or parallel set-up in script?: False

Who can help

@mfuntowicz (based on initial commit of convert_graph_to_onnx) @stas00 (based on model used here) @thomwolf (based on history)

Information

Model I am using (Bert, XLNet …): facebook/wmt19-en-de

The problem arises when using:

• the official example scripts: transformers.convert_graph_to_onnx.convert
• my own modified scripts: (give details below)

The tasks I am working on is:

• an official GLUE/SQUaD task: (give the name)
• my own task or dataset: converting the translation model to onnx

To reproduce

Steps to reproduce the behavior:

import torch
import transformers
from transformers import convert_graph_to_onnx
from pathlib import Path

nlp = transformers.pipeline("translation_en_to_de", model="facebook/wmt19-en-de", tokenizer="facebook/wmt19-en-de")
convert_graph_to_onnx.convert(
    framework="pt",
    model="facebook/wmt19-en-de",
    output=Path("encoder/en_de_trans.onnx"),
    opset=12,
    tokenizer="facebook/wmt19-en-de",
    use_external_format= False,
    pipeline_name= "translation_en_to_de",
)

Raises:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1-d46bec961b86> in <module>
      5 
      6 nlp = transformers.pipeline("translation_en_to_de", model="facebook/wmt19-en-de", tokenizer="facebook/wmt19-en-de")
----> 7 convert_graph_to_onnx.convert(
      8     framework="pt",
      9     model="facebook/wmt19-en-de",

~/anaconda3/envs/dev/lib/python3.8/site-packages/transformers/convert_graph_to_onnx.py in convert(framework, model, output, opset, tokenizer, use_external_format, pipeline_name)
    365     # Export the graph
    366     if framework == "pt":
--> 367         convert_pytorch(nlp, opset, output, use_external_format)
    368     else:
    369         convert_tensorflow(nlp, opset, output)

~/anaconda3/envs/dev/lib/python3.8/site-packages/transformers/convert_graph_to_onnx.py in convert_pytorch(nlp, opset, output, use_external_format)
    274 
    275     with torch.no_grad():
--> 276         input_names, output_names, dynamic_axes, tokens = infer_shapes(nlp, "pt")
    277         ordered_input_names, model_args = ensure_valid_input(nlp.model, tokens, input_names)
    278 

~/anaconda3/envs/dev/lib/python3.8/site-packages/transformers/convert_graph_to_onnx.py in infer_shapes(nlp, framework)
    196     tokens = nlp.tokenizer("This is a sample output", return_tensors=framework)
    197     seq_len = tokens.input_ids.shape[-1]
--> 198     outputs = nlp.model(**tokens) if framework == "pt" else nlp.model(tokens)
    199     if isinstance(outputs, ModelOutput):
    200         outputs = outputs.to_tuple()

~/anaconda3/envs/dev/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

TypeError: forward() got an unexpected keyword argument 'token_type_ids'

Subsequently, the call of the raise can be boiled down to inferring the shapes for torch.onnx.export

I think that may be due to the incompatibility of the tokenizer() vs tokenizer.encode() for this very model.

import transformers
tokenizer = transformers.AutoTokenizer.from_pretrained("facebook/wmt19-en-de")
model = transformers.AutoModelForSeq2SeqLM.from_pretrained("facebook/wmt19-en-de")
string = "Hello. How are you?"

# model.generate(tokenizer(string, return_tensors="pt")) # Fails

model.generate(tokenizer.encode(string, return_tensors="pt")) # Succeeds

Expected behavior

Model export should work properly.

[oborchers]

Thank you for your help, @stas00! I followed your advice and created a new issue.

[stas00]

The problem you reported has been fixed in https://github.com/huggingface/transformers/pull/9736 (merged already)

But then another one poped up in https://github.com/huggingface/transformers/issues/9737

You can just use the https://github.com/huggingface/transformers/pull/9738 branch - since it contains both fixes.

Not sure how quickly it will get merged, since we might want to solve this for other models too. I made only a local for fsmt fix in that PR.