InferenceServerClient request time is large than model-analyzer

Description I train a resnet50 mdoel with torch, then model.pt–>model.onnx–>model.trt and I use model-analyzer to test model, the result follow:

For resnet_trt_config_default ,p99 Latency is 48.3ms.

Last I run the model with resnet_trt_config_default config :

docker run --rm -p 8000:8000 -p 8001:8001 -p 8002:8002 -v  /model_rep:/models nvcr.io/nvidia/tritonserver:22.07-py3 tritonserver --model-repository=/models

request the server

start_time=time.time()
results = triton_client.infer('resnet_pt', inputs=inputs, outputs=outputs)    
print("infer time:",time.time()-start_time)

# **infer time: 602.1ms**

the infer time is too large

Triton Information image: nvcr.io/nvidia/tritonserver:22.07-py3 cuda: 11.6 config:

name: "resnet_trt"
platform: "tensorrt_plan"
max_batch_size: 1
input {
  name: "input"
  data_type: TYPE_FP32
  dims: 3
  dims: 297
  dims: 640
}
output {
  name: "output"
  data_type: TYPE_FP32
  dims: 2
}
instance_group {
  count: 6
  kind: KIND_GPU
}
dynamic_batching {
}
backend: "tensorrt"

Expected behavior why infer time is large than model-analyzer results? if you can give some advice,very thanks!

[Lzhang-hub]

Sure! You can use the metrics API to get queue time, infer duration, and a lot more: https://github.com/triton-inference-server/server/blob/main/docs/user_guide/metrics.md

For debugging purposes (not production), tracing your requests can also get exact timestamps: https://github.com/triton-inference-server/server/blob/main/docs/protocol/extension_trace.md

Thanks! I will try it

[dyastremsky]

My pleasure! Good luck. Closing ticket now. If you need further assistance with this issue, let me know and I’ll reopen it.