Performance on triton with python backend

Description Hello

• I was expecting that triton would be better than torchserve when I read some documentation. But in the images bellow, i see that: with the same model, the throughput of triton is lower the throughput of torchserve. Despite of I used tensorrt model in triton (the throughput of tensorrt model inference are faster than pytorch model inference)

• Could it be that was wrong at some point ?

Triton Information

Version triton 21.07

To Reproduce

This is what i did:

• In model repository folder, with using python backend: I add a model in model repos (creating model.py file, adding the weight of the tensorrt trained model here, creating config.pbtxt file, build stub and conda-pack my envs, …)
• In model.py at TritonPythonModel class:
  • I load pretrained model at initialize function
  • In excuse funcion, I implement my model inference and convert to format for triton response.

Expected behavior

Am I doing it right?. If I’m wrong can you point me in the right direction.

[Tabrizian]

@gioipv Could you please share your model.py file. From your overall description, it seems like you are doing it correctly. What inferencing solution does “model inference” column use? Is it using the PyTorch backend?

How did you measure the performance of your Python models? Did you use Perf Analyzer?

[Tabrizian]

@gioipv Thanks for sharing the results.

What do you think about that: (triton’s client API can take a long time)

Triton Python Client may add some latency because of using Python API which could be slower. Also, if you want to try concurrency values higher than 1 it would be harder to create the same scenario using Python client.