Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

cache per-model dependency in model archiver

See original GitHub issue

🚀 The feature

Correct me if I’m wrong: Currently torchserve supports per-model dependency where user can specify --requirements-file when running torch-model-archiver. and when model gets loaded, torchserve creates some sort of python venv to install these extra packages.

To reduce the wait time, is it possible to cache the dependency packages somehow so that when the model gets loaded, it can be ready for serving (almost) immediately without waiting for several minutes to install dependencies?

Motivation, pitch

see above

Alternatives

No response

Additional context

No response

Issue Analytics

State:
Created a year ago
Comments:6 (2 by maintainers)

Top GitHub Comments

1reaction

hgong-snapcommented, Oct 31, 2022

Yeah that looks about right, I will caution you though in that I haven’t seen people do this just because then the provenance of venv is funky and your model packaging will take longer.

Also I believe os.system() will spawn a seperate process so you want the activation to happen within the same python interpreter so maybe this helps https://stackoverflow.com/a/14792407

thanks this is really helpful! will try!

0reactions

hgong-snapcommented, Dec 10, 2022

actually, one question related to this:

suppose model specified my_package in requirements.in, but it is already installed in base image(aka. the requirement is already satisfied), will my_package re-installed for this model? or will skip and use this package in base image ?

Top Results From Across the Web

Torch Model archiver for TorchServe - GitHub

A key feature of TorchServe is the ability to package all model artifacts into a single model archive file. It is a separate...

PyTorch - KServe Documentation Website

TorchServe provides a utility to package all the model artifacts into a single Torchserve Model Archive Files (MAR). You can store your model...

6. Custom Service — PyTorch/Serve master documentation

Custom handlers. Creating model archive with entry point. Handling model execution on GPU. Installing model specific python dependencies ...

Release Notes for OpenVINO™ toolkit v.2022.2 - Intel

Implemented model caching support. The feature allows to significantly improve first inference latency. Improved inference performance for non-vision use cases ...

Deploying PyTorch models for inference at scale using ...

You can use the torch-model-archiver tool in TorchServe to create a .mar ... number of workers per model, describing the status of a...