deploying model using trtis is much slower than using frozen model directly

after optimizing yolov3 using tf-tensorrt optimization, i infer in two way.

1. using trtis (~20fps) + bellow warning

layout failed: Invalid argument: The graph is already optimized by layout optimizer.

1. using frozen directly (~54 fps)

[deadeyegoodwin]

Are you actually providing a labels.txt? Given the dimension of those outputs the file would need to have 10647*80=850k+ entries… but that isn’t related to your issue.

So you are using the offline TF-TRT conversion to create a graphdef, and then running that in TRTIS and also using some script that uses the TF API to load and run it directly?

What version of TRTIS are you using? Have you tried using perf_client with TRTIS and see what performance that gives you for various concurrency levels? Make sure you read this section of the documentation: https://docs.nvidia.com/deeplearning/sdk/tensorrt-inference-server-guide/docs/optimization.html

[deadeyegoodwin]

Closing. Please reopen with perf_client results if those still show a problem