Dynamic quantized ORTModelForSeq2SeqLM throws error during inference

System Info

Optimum - 1.4.1
Linux
Python - 3.7

Who can help?

@JingyaHuang @echarlaix I dynamically quantized a model fine-tuned on T5 for the text-to-text generation task, but it was showing error during inference.

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, …)
• My own task or dataset (give details below)

Reproduction

I tried to load that quantized model using the following code:

model_ort_q = ORTModelForSeq2SeqLM.from_pretrained("local_folder_path")

First it complained that file was not found. Even If I pass the file_name, it was showing the below error:

NoSuchFile: [ONNXRuntimeError] : 3 : NO_SUCHFILE : Load model from ./ort-quant-fin/encoder_model.onnx failed:Load model ./local_path/encoder_model.onnx failed. File doesn't exist

So I renamed the file by removing the suffix “_quantized” in all the onnx model filenames. Then the model got loaded successfully.

But when I tried to do an inference, I got the below error:

text2text_generator = pipeline("text2text-generation", model=model_ort, tokenizer=tokenizer)
print(text2text_generator("some text"))

[/usr/local/lib/python3.7/dist-packages/optimum/onnxruntime/modeling_seq2seq.py](https://localhost:8080/#) in forward(self, input_ids, attention_mask, **kwargs)
    530     ) -> BaseModelOutput:
    531 
--> 532         onnx_inputs = {"input_ids": input_ids.cpu().detach().numpy()}
    533 
    534         # Add the attention_mask inputs when needed

AttributeError: 'NoneType' object has no attribute 'cpu'

Expected behavior

The quantized model for ORTModelForSeq2SeqLM should generate text during inference

[leslyarun]

@JingyaHuang Awesome works now

[JingyaHuang]

Hi @leslyarun ORTModel needs the model configuration(config.json) instead of the configuration of your quantization approach(ort_config.json). You can save the model config when loading the original model:

model = ORTModelForSeq2SeqLM.from_pretrained(model_id, from_transformers=True)
model.save_pretrained(onnx_path)