Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

TorchVision support for pre-trained, quantised, ResNet50 using QNNpack backend

See original GitHub issue

🚀 Feature

Addition of pre-trained, quantised ResNet50 model to torchvision.models with support for QNNpack backend.

Motivation

Quantised models have been shown to deliver greater performance than their fp32 equivalents without sacrificing accuracy. The quantised ResNet50 model currently available in TorchVision supports the FBGEMM backend, but does not provide support for QNNpack. On AArch64, only the QNNpack backend is currently supported, so expanding the functionality of the ResNet50 model would allow its use one AArch64 platforms. Expanding the capabilities of the quantised models available to TorchVision users would provide functionality equivalent to what is currently available in other frameworks.

Pitch

Quantised, pre-trained, ResNet50 models are available for use on with TensorFlow (https://github.com/IntelAI/models/blob/master/benchmarks/image_recognition/tensorflow/resnet50v1_5/README.md#int8-inference-instructions) and typically deliver improved performance over their fp32 equivalents.

Are there any plans to expand the supported backends for the quantised ResNet50 model to TorchVision to cover QNNpack?

Alternatives

Where the workload or benchmark requires ResNet50, the alternative is to make your own quantised model using the qnnpack backend.

Issue Analytics

State:
Created 3 years ago
Reactions:1
Comments:5 (3 by maintainers)

Top GitHub Comments

2reactions

nSircombecommented, Feb 9, 2021

Thanks @oke-aditya, I didn’t find any details of QNNpack support for ResNet50 in the plans, but have asked in-thread.

I’ve updated my ‘Feature Request’ above in light of our discussion, so it’s more specific to my use-case (QNNpack backend, on AArch64).

0reactions

fmassacommented, Aug 13, 2021

Thanks for the request and sorry for the delay in replying.

I might not remember correctly anymore, but IIRC we provided fbgemm backends for ResNet because fbgemm was better suited for server inference, and ResNet50 was normally too big for running on embedded devices (on which qnnpack was better).

@nSircombe are you planning on running ResNet50 on embedded devices, or is it that fbgemm doesn’t work in your system?

Top Results From Across the Web

Quantization Recipe - PyTorch Tutorials

Quantization Recipe. This recipe demonstrates how to quantize a PyTorch model so it can run with reduced size and faster inference speed with...

resnet50 — Torchvision main documentation - PyTorch

Note that quantize = True returns a quantized model with 8 bit weights. Quantized models only support inference and run on CPUs. GPU...

Pytorch object detection model optimization - Stack Overflow

But when i tried to quantize frcnn model, got the following error, "RuntimeError: Quantized backend not supported" model_quantized= torchvision.

Borealis AI Quantization Notebook - Kaggle

Explore and run machine learning code with Kaggle Notebooks | Using data from Cassava ... backend not supported ") torch.backends.quantized.engine = backend ......

Quantizing Resnet50 - NVIDIA Documentation Center

This package provides a number of quantized layer modules, which contain quantizers ... Starting with a pre-trained model, the simplest way to do...