Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

ONNX export (for transformer)

See original GitHub issue

Hi,

I’ve seen that you have started to work on ONNX export. I tried it on my own for the transformer model but have issues here:

File "fairseq/fairseq/models/fairseq_model.py", line 160, in forward
    decoder_out = self.decoder(prev_output_tokens, encoder_out)
[...]
ValueError: Auto nesting doesn't know how to process an input object of type dict. Accepted types: Tensors, or lists/tuples of them

The issue is that “encoder_out” is a dictionary. Any idea how to solve that in an elegant way (other than converting to list)? 😃 Or will be more support from pytorch soon?

Update: Conversion is not possible right now since “expand_as” and “ne” are not implemented/merged yet (see https://github.com/pytorch/pytorch/issues/10882 and https://github.com/pytorch/pytorch/pull/10914).

Cheers!

Issue Analytics

State:
Created 5 years ago
Comments:12 (7 by maintainers)

Top GitHub Comments

11reactions

vipulbjjcommented, Jun 19, 2020

Unable to cast from non-held to held instance (T& to Holder<T>) (compile in debug mode for type information)

I’m getting this error after adding ONNX_ATEN_FALLBACK as mentioned by @zrphercule . Did anyone solve this?

3reactions

jamesr66acommented, Dec 6, 2018

Hi everyone,

It is in fact the case that the Transformer model is not supported by pure-ONNX export, but rather it is exportable via the ONNX-ATen fallback path. For an example of how that is enabled, see this PR: https://github.com/pytorch/pytorch/pull/14492. With ONNX ATen fallback, an ONNX backend that supports running ops from the ATen library (i.e. PyTorch’s library) can interpret these ops and run the original implementations directly.

The reason this is necessary is that the ONNX specification does not support many operators that are used in modern PyTorch code, such as the Transformer network. Adding operators to ONNX involves a lengthy approval process, and requires writing code for both ONNX front-ends and back-ends. So, to provide a solution that works earlier, this fallback path is available.

cc @zrphercule do you want to take a look at the ops missing for this use case and make sure they’re on the roadmap for addition?

Top Results From Across the Web

Export to ONNX - Transformers - Hugging Face

In this guide, we'll show you how to export Transformers models to ONNX (Open Neural Network eXchange). Once exported, a model can be...

Convert Transformers to ONNX with Hugging Face Optimum

There are currently three ways to convert your Hugging Face Transformers models to ONNX. In this section, you will learn how to export...

04-onnx-export - Jupyter Notebooks Gallery

Exporting transformers model to ONNX. Exporting models (either PyTorch or TensorFlow) is easily achieved through the conversion tool provided as part of ...

NLP Transformers pipelines with ONNX - Towards Data Science

ONNX is a machine learning format for neural networks. It is portable, open-source and really awesome to boost inference speed without sacrificing accuracy....

Convert your bulky Transformer models into lightweight high ...

“ ALBERT model has shared weights among layers as part of the optimization from BERT . The export torch.onnx.export outputs the weights to...