RAM Usage

Hi, I noticed that TorchServe takes up a lot of RAM. This is unexpected, especially as the models run on GPU and take up space there too. In my particular case, I have 2 GPUs and 3 models for a particular inference chain, and nvidia-smi shows that the 3 models x 2 GPUs take approximately 6 GB on the GPUs (i.e. ~ 1 GB per model per GPU). At the same time, htop shows that they are also occupying ~ 12 GB of RAM in total. This should be close to zero as far as I understand. Can anyone please help me understand / resolve this issue?

[cgnhmt]

Hi does anyone still having this problem?

I have 7 models and serving with the TorchServe. If I load all of them to CPU, in total they consume around 3 gb of RAM. But if I load on GPU, they are using excess amount of the RAM. Only one model consumes around 3gb RAM and If I load all on the GPU they take around 18-20gb. Another interesting thing is that after making an inference, used RAM increases dramatically and stays there. Here is the story:

• There are 7 models, some of them works paralel. Only one is loaded on GPU.
• Before an inference, total RAM usage is 3gb
• During and after inference, total RAM usage is 12gb

Maybe I am doing something wrong with the configurations. I can provide more details.

Here is my config file inference_address=http://0.0.0.0:8080 management_address=http://0.0.0.0:8081 metrics_address=http://0.0.0.0:8082 model_store=/home/model-server/model-store install_py_dep_per_model=false load_models=model1,model2,model3,…,model7 default_workers_per_model=1

Ps. I am using torchserve version 0.4.2 but I tried with the 0.5.2 and it was still same.

Thank you for the help!

[lxning]

this is fixed in v0.4.0