No performance improvement when optimizing models

Description In config.pbtxt file I specify TensorRT optimization but inference performance is the same.

Triton Information 21.05-py3 Running pre-built docker container.

To Reproduce Model config.pbtxt:

name: "VGG16"
platform: "tensorflow_savedmodel"
max_batch_size: 64
input  {
    name: "Input"
    data_type: TYPE_FP32
    dims: [ 224, 224, 3 ]
    format: FORMAT_NHWC
}
output  {
    name: "VGG16"
    data_type: TYPE_FP32
    dims: [ 1000]
    is_shape_tensor: false
}
optimization { execution_accelerators {
  gpu_execution_accelerator : [ {
    name : "tensorrt"
    parameters { key: "precision_mode" value: "FP16" }}]
}}

Expected behavior If I use TensorRT optimization manually / outside de Triton Server container, inference speed improves by an order of magnitude. I expect the same to happen by loading the model in Triton Server.

params = trt.DEFAULT_TRT_CONVERSION_PARAMS._replace(precision_mode='FP32')
converter = trt.TrtGraphConverterV2(
   input_saved_model_dir = self.model_dir,
   conversion_params = params)
converter.convert()

[deadeyegoodwin]

The most reliable path is to apply TF-TRT optimization outside of triton and then use the resulting TF model with triton. If you do that you should see the full performance improvement provided by the TF-TRT optimization. Using TF-TRT optimization “online” in triton is less reliable (as you have seen). When doing the optimization offline be sure to request fp16 precision if that is what you want (as you did in the online specification)

[deadeyegoodwin]

Closing due to inactivity.