nv_gpu_memory_used_bytes metric does not decrease on model unload

From local testing with 20.01 the metric nv_gpu_memory_used_bytes exposed on /metrics does not decrease on model unload. Assuming this is expected would there be some way to expose the actual memory used by the loaded models?

I ask as orchestrators (like when running inside kubernetes) might wish to determine the available memory for new models on the server.

[deadeyegoodwin]

Any deployment plan should be aware of the limitations of the different backends. TF is not a particularly good framework for inference for a couple of reasons (this memory usage policy being one of them). If you want to have a long-running “dynamic model repository” TRTIS instance where you use the model-control APIs to load and unload models, then you need to account for TF. TRTIS provides a command-line option to limit TF to a fraction of the GPU memory… but this doesn’t cause it to release its memory.

For a system like kubernetes an alternative is to use a “static model repository” TRTIS instance where TRTIS is started/initialized with some set of models and that set of models never changes. If a change is needed then use k8s to change configuration via a rolling update or similar.

Each approach has advantages and disadvantages.

[cliveseldon]

Thanks for your feedback. Very useful.