Onnx T5 for Generation

Environment info

adapter-transformers version: 2.1.2

• Platform: Windows-10-10.0.19041-SP0
• Python version: 3.7.5
• PyTorch version (GPU?): 1.8.1+cpu (False)
• Tensorflow version (GPU?): 2.3.0 (False)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: No
• Using distributed or parallel set-up in script?: No

Who can help

@patrickvonplaten, @patil-suraj

Information

I want to use the to Onnx converted T5 model for generation, but I can only pass decoder_input_ids with a sequence length of 1.

To reproduce

Steps to reproduce the behavior:

1. Convert the T5 model to onnx: python -m transformers.onnx --model=t5-base --feature=seq2seq-lm onnx/t5-base/
2. Load the onnx model with onnxruntime: session = onnxruntime.InferenceSession('onnx/t5-base/model.onnx')
3. Pass the model an input with a decoder sequence with more than one element:

tokenizer = AutoTokenizer.from_pretrained("t5-base")
encoder_input = tokenizer("This is some text.", return_tensors="np")
decoder_inputs = tokenizer("bla bla", return_tensors="np")
print(decoder_inputs)

model_input = {
    "input_ids": encoder_input["input_ids"], 
    "attention_mask": encoder_input["attention_mask"], 
    "decoder_input_ids": decoder_inputs["input_ids"], 
    "decoder_attention_mask": decoder_inputs["attention_mask"]
}
outputs = session.run([], model_input)

Expected behavior

I would expect there to be a way to pass multiple decoder_input_ids to the model to generate text. How is this intended to be done?

[hSterz]

That is because for now it is expected to be used with past_key_values, meaning that only the last decoder_input_ids are neeeded. We are currently working on changing this and it should be merged by the end of the year.

I see. What would I use for the first token that is generated as past key values?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.