onnx run error at translation model

System Info

• transformers version: 4.17.0
• Platform: Linux-5.4.0-122-generic-x86_64-with-debian-bullseye-sid
• Python version: 3.7.11
• PyTorch version (GPU?): 1.8.1 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: <fill in>
• Using distributed or parallel set-up in script?: <fill in>

Who can help?

@patrickvonplaten @patil-suraj

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, …)
• My own task or dataset (give details below)

Reproduction

1、convert model to onnx： python3 -m transformers.onnx --model opus-mt-en-zh --atol=2e-04 --feature=seq2seq-lm opus-mt-en-zh-onnx-301 tips： Validating ONNX model… -[✓] ONNX model output names match reference model ({‘logits’}) - Validating ONNX Model output “logits”: -[✓] (2, 8, 65001) matches (2, 8, 65001) -[✓] all values close (atol: 0.0002) All good, model saved at: opus-mt-en-zh-onnx-301/model.onnx

2、translation：

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from onnxruntime import InferenceSession

tokenizer=AutoTokenizer.from_pretrained("opus-mt-en-zh")
session = InferenceSession("opus-mt-en-zh-onnx-301/model.onnx")
inputs = tokenizer("Using DistilBERT with ONNX Runtime!", return_tensors="pt")
outputs = session.run(output_names=["last_hidden_state"], input_feed=dict(inputs))

tips：

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/xieyouxi/anaconda3/envs/HuggingFace-torch-gpu/lib/python3.7/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 196, in run
    raise ValueError("Model requires {} inputs. Input Feed contains {}".format(num_required_inputs, num_inputs))
ValueError: Model requires 4 inputs. Input Feed contains 2

Expected behavior

Unable to translate from en to zh, Am I using the wrong interface？

[regisss]

@xyx361100238 The error message says that the model requires 4 inputs but you are providing only 2 of them. Either you need to provide the missing inputs, or you need to modify the OnnxConfig associated to your model to specify only 2 inputs.

The architecture of opus-mt-en-zh seems to be MarianMTModel. According to what I see in the OnnxConfig here, the 4 expected inputs are:

• input_ids,
• attention_mask,
• decoder_input_ids,
• decoder_attention_mask.

So I think you are only providing input_ids and attention_mask here. To generate the missing inputs, you can take a look at how dummy inputs used for exporting the model are generated.

[michaelbenayoun]

So you also need to provide the inputs for the decoder side, something along the lines:

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from onnxruntime import InferenceSession

tokenizer=AutoTokenizer.from_pretrained("opus-mt-en-zh")
session = InferenceSession("opus-mt-en-zh-onnx-301/model.onnx")
inputs = tokenizer("Using DistilBERT with ONNX Runtime!", return_tensors="pt")
inputs["decoder_input_ids"] = torch.tensor([tokenizer.bos_token_id], dtype=torch.long)
inputs["decoder_attention_mask"] = torch.tensor([1], dtype=torch.long)
outputs = session.run(output_names=["last_hidden_state"], input_feed=inputs)

What is true is that it would be easier if the decoder_attention_mask was automatically generated, but we you currently need to provide it manually.