Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

How to cache inferences with torchserve

See original GitHub issue

Reference architecture showcasing how to cache inferences from torchserve

So potentially the inference handler would reach from some cloud cache or KV store

The benefit of this is it’d dramatically reduce latency for common queries

Probably a good level 3-4 bootcamp task for a specific kind of KV store like Redis or specific cloud cache in AWS.

Issue Analytics

State:
Created 2 years ago
Comments:9 (4 by maintainers)

Top GitHub Comments

1reaction

msaroufimcommented, Oct 6, 2021

Closing for now since this could be handled outside of torchserve more easily

1reaction

msaroufimcommented, Sep 18, 2021

That is correct, if you have a reference example we can add the example for it in the repo

Top Results From Across the Web

12. Running TorchServe — PyTorch/Serve master ...

TorchServe can be used for many types of inference in production settings. It provides an easy-to-use command line interface and utilizes REST based...

Serving PyTorch models with TorchServe | by Álvaro Bartolomé

TorchServe is the ML model serving framework developed by PyTorch. This post explains how to train and serve a CNN transfer learning model....

BERT TorchServe Tutorial — AWS Neuron Documentation

This tutorial demonstrates the use of TorchServe with Neuron, ... Download the custom handler script that will eventually respond to inference requests.

PyTorch - KServe Documentation Website

In this example, we use a trained pytorch mnist model to predict handwritten digits by running an inference service with TorchServe predictor.

Deploying EfficientNet Model using TorchServe

For more information on batch inference with TorchServe, please refer to ... cache/b4.pt --handler handler.py \ --export-path model-store ...