Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Allow selected models in the model repository not to be loaded

See original GitHub issue

Is your feature request related to a problem? Please describe. Yes. Some models in my model repository runs just on CPU and some others just GPU. If launching the server without GPU support, it fails because backends&models with GPU support fails to load. Would like to be able to exclude some models being loaded in case not launching triton with GPU support. Describe the solution you’d like

Maybe naming the model folders with a prefix of “". So all model folder names in a repo starting with "” character will be omitted to be loaded.
It will skip loading failed models and without crashing, continue with successful models.

Describe alternatives you’ve considered Currently, if I want to test CPU only models, I move GPU models to temporary folder.

Issue Analytics

State:
Created 2 years ago
Comments:7 (4 by maintainers)

Top GitHub Comments

2reactions

tanmayv25commented, Sep 3, 2021

@mustafakasap You can try out --exit-on-error=false when launching server. See here for more details: https://github.com/triton-inference-server/server/blob/main/src/servers/main.cc#L310

0reactions

mustafakasapcommented, Sep 3, 2021

Top Results From Across the Web

Getting Started with Repositories

This document shows how to handle repositories through the web interface as well as through the terminal. There are no requirements if working...

MLflow Model Registry on Databricks

MLflow Model Registry is a centralized model repository and a UI and set of APIs that enable you to manage the full lifecycle...

Model Repository: DataRobot docs

These blueprints illustrate the algorithms (the preprocessing steps, selected estimators, and in some models, postprocessing as well) used to build a model, not...

With the SageMaker model registry you can catalog models for production, ... Model packages used in the AWS Marketplace are not versionable entities...

Develop models in Code Repositories

If configured, a model is automatically submitted whenever a transaction is successfully committed to that model. However, as it is not always desired...