model expected the shape of dimension 0 to be between 1 and 1 but received 32

I want to use dynamic batching in triton plan model. I use pytorch hub resnet50 pretrained model and convert to onnx.

Model input and output information

Then I use trtexec to convert onnx model into tensorrt.

trtexec --onnx=resnet50_onnx_bs.onnx \
--saveEngine=model.plan \
--explicitBatch \
--minShapes=input:1x3x224x224 \
--optShapes=input:8x3x224x224 \
--maxShapes=input:32x3x224x224

In order to deploy model at triton server, I follow the instructions and directory is

Config file

Start triton server

Client code

import time
import numpy as np
import tritonclient.http as httpclient
from PIL import Image

if __name__ == "__main__":
    triton_client = httpclient.InferenceServerClient(url=127.0.0.1:8000')

    image = Image.open('./dog.jpg')
    image = image.resize((224, 224), Image.ANTIALIAS)
    image = np.asarray(image)
    image = image / 255
    image = np.expand_dims(image, axis=0)
    image = np.array(np.repeat(image, 32, axis=0), dtype=np.float32)
    image = np.transpose(image, axes=[0, 3, 1, 2])
    image = image.astype(np.float32)
    print("input shape: ", image.shape)  # 32,3,224,224
    inputs = []
    inputs.append(httpclient.InferInput('input', image.shape, "FP32"))
    inputs[0].set_data_from_numpy(image, binary_data=False)
    outputs = []
    outputs.append(httpclient.InferRequestedOutput('output', binary_data=False)) 
    results = triton_client.infer('trt-resnet50', inputs=inputs, outputs=outputs)

Unfortunately, I get error information.

root@sib-1080Ti_x2-225:/models# python client.py
client.py:12: DeprecationWarning: ANTIALIAS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
  image = image.resize((224, 224), Image.ANTIALIAS)
input shape:  (32, 3, 224, 224)
Traceback (most recent call last):
  File "client.py", line 58, in <module>
    results = triton_client.infer('trt-resnet50', inputs=inputs, outputs=outputs)
  File "/usr/local/lib/python3.8/dist-packages/tritonclient/http/__init__.py", line 1372, in infer
    _raise_if_error(response)
  File "/usr/local/lib/python3.8/dist-packages/tritonclient/http/__init__.py", line 64, in _raise_if_error
    raise error
tritonclient.utils.InferenceServerException: request specifies invalid shape for input 'input' for trt-resnet50. Error details: model expected the shape of dimension 0 to be between 1 and 1 but received 32

I use similar configuration for pytorch/onnx/openvino, but they can output result successfully. My questions are

1. Does tensorrt support dynamic batch and how to solve above error.
2. Can I set “execution_accelerators” in onnx config as an alternative. Something like this.

optimization { execution_accelerators {
  gpu_execution_accelerator : [ {
    name : "tensorrt"
    parameters { key: "precision_mode" value: "FP16" }
    parameters { key: "max_workspace_size_bytes" value: "1073741824" }
    }]
}}

[wanduoz]

The issue is coming from the model itself. I think in your scenario you don’t need to use explicit batch and should build the model with implicit batch. (cc @tanmayv25 to correct me if I’m wrong here).

Thanks for replying. I try the belowing command

trtexec --onnx=resnet50_onnx_bs.onnx \
--saveEngine=model.plan \
--minShapes=input:1x3x224x224 \
--optShapes=input:8x3x224x224 \
--maxShapes=input:32x3x224x224

Then I use the same client code, but still get dimension error

tritonclient.utils.InferenceServerException: request specifies invalid shape for input 'input' for trt-resnet50. Error details: model expected the shape of dimension 0 to be between 1 and 1 but received 32

(By the way, I don’t know how cc someone. Do you mean @ someone?)

---

There are still questions bothering me.

1. When I use onnx or torchScript model in triton, Can I just set optimization tensorrt and the triton will conver model into tensorrt plan file(the 2nd question I asked above)? If so what is the difference between manually convert model and automatically convert model?
2. Back to topic of dynamic batching, can I just batch data at client side and send it to server or I need to simulate concurrent scenario(start 32 processes and each send one image data)? Can triton handle both?
3. I want to estimate the inference performance and I calculate time interval before and after infer. But It cose about 4s. Do you have any idea about this?

start = time.time()
results = triton_client.infer('trt-resnet50', inputs=inputs, outputs=outputs)
end = time.time()
print("triton trt infer: ", (end-start), "s")

I didn’t test on tensorrt because of dimension error but it works for torchScript/tf/onnx.

[tanmayv25]

Closing as the original issue is resolved.