Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Does model should instantly free memory after unload?

See original GitHub issue

I tested the model repository protocol, which allows to load and unload models. I was sequentially called load and unload requests, like:

POST v2/repository/models/${MODEL_NAME}/load
POST v2/repository/models/${MODEL_NAME}/unload
POST v2/repository/models/${MODEL_NAME}/load
POST v2/repository/models/${MODEL_NAME}/unload
POST v2/repository/models/${MODEL_NAME}/load
POST v2/repository/models/${MODEL_NAME}/unload
POST v2/repository/models/${MODEL_NAME}/load
POST v2/repository/models/${MODEL_NAME}/unload

Every time I loaded and unloaded the same model with the same version. Nothing was changed in the repository at this time. The model backend was PyTorch, model is loaded on Host memory (CPU Instance) I guessed that the total used memory would not change after the first load, but actually, that is not true. After every unload triton free a few memory, much smaller than was got on load, on each load triton takes more and more host memory. So in a log way, it may cause Memory Error. Is that undefined behavior?

Issue Analytics

State:
Created a year ago
Comments:5 (4 by maintainers)

Top GitHub Comments

1reaction

alxmamaevcommented, Aug 15, 2022

@dyastremsky Im not shure about version, because I experimented with different containers versions, I’ll check and give you actually feedback .

0reactions

dyastremskycommented, Aug 31, 2022

Closing due to inactivity. Let us know if you need follow-up and we can reopen the issue.

Top Results From Across the Web

Unity Memory Management: Unload That Asset!

I have a question for you regarding Unity memory management; when exactly is Unity releasing the memory your game assets are taking?

Clearing Tensorflow GPU memory after model execution

But it doesn't unload memory when it's finished. When the second model is loaded, using both tf.reset_default_graph() and with tf.Graph().

Memory management | Addressables | 1.3.8 - Unity - Manual

To unload the asset, use the Addressables.Release method, which decrements the ref-count. When a given asset's ref-count is zero, that asset is ready...

How could I release gpu memory of keras - Part 1 (2017)

I ran into similar problems. I found that deleting the model variable and calling the gc.collect() helped. The behaviour of GPU memory getting ......

Force Editor to release memory? - Unity Forum

Also I was slightly mistaken. Managed Heap Memory can be returned to the OS if the entire heap section has been emptied. It's...