Python backend bug when using model load api to reload model in explicit mode

Description I have a found a bug when using the model load api to re-load an already loaded python backend model in explicit mode. To explain the scenario, We have a model which is already being loaded and running by Triton. We have a new version of the model which we would like Triton to load. So when triton is currently running and serving the model, we deleted the same model from model repository and copied the new version of the model. If we call the load API /v2/repository/models/<model_name>/load it will start re-loading the model. The bug here is that triton is not using the conda environment of the new version of the model. it continues to use the conda environment of old model hence resulting is failed model re-load in case there are newer dependencies in the conda environment for newer version.

Triton Information What version of Triton are you using? r22.07

Are you using the Triton container or did you build it yourself? Triton Container

To Reproduce Steps to reproduce the behavior. Its easy to reproduce the issue, create two versions of same python backend model. one with an additional python dependency i.e both versions just only differ in conda environments (provided example model.py and requirements.txt below for each version). use triton to load one version of the model and when triton is successfully loaded the model and running, remove the model from model repository and copy the other version which has an additonal dependency and call the model reload api. it will fail saying module not found error(for example, in the below case it will fail to re-load with pandas module not found error).

Version 1 - model.py

import triton_python_backend_utils as pb_utils
 
import numpy as np
 
 class TritonPythonModel:
    def initialize(self, args):
    	pass
    def execute(self, requests):
    	pass
    def finalize(self):
        pass

version 1 - requirements.txt

numpy

Version 2 - model.py

import triton_python_backend_utils as pb_utils
 
import numpy as np
import pandas as pd
 
 class TritonPythonModel:
    def initialize(self, args):
    	pass
    def execute(self, requests):
    	pass
    def finalize(self):
        pass

version 2 - requirements.txt

numpy
pandas

Describe the models (framework, inputs, outputs), ideally include the model configuration file (if using an ensemble include the model configuration file for that as well). python backend

Expected behavior A clear and concise description of what you expected to happen. when reloading an model it should reload conda environment too

[SaratM34]

Hi @dyastremsky Thanks so much for the update.

[dyastremsky]

Thanks for checking in. No update yet. When there is an update and/or corresponding pull request, this GitHub issue should be updated as well.