Warning: Explicit batch network detected and batch size specified, use enqueue without batch size instead.

Description I created a Tensor RT plan using NetworkDefinitionCreationFlags::kEXPLICIT_BATCH, because the current master branch of the TensorRT-ONNX Parser does not allow implicit batch dimension. But there are no wildcards in any of the input dimensions.

TRTIS 19.10 can load the model, but on every inference the TensorRT Warning Warning: Explicit batch network detected and batch size specified, use enqueue without batch size instead. is logged. I suppose this is because IExecutionContext::enqueue is used instead of IExecutionContext::enqueueV2. Is this expected behavior, or am I doing something wrong?

Side question: Disregarding the warning, will this carry a performance penalty? Should I try to make my model use implicit batch dimensions, when not using dynamic input shapes?

TRTIS Information What version of TRTIS are you using? 19.10

Are you using the TRTIS container or did you build it yourself? NGC container 19.10-py3

To Reproduce Steps to reproduce the behavior: Create model with NetworkDefinitionCreationFlags::EXPLICIT_BATCH

Expected behavior No warning is emitted.

[mrjackbo]

Is there a reason you are using a fixed batch dimension in your model?

Yes, I misused the current version of NvOnnxParser. Now I managed to convert my model to explicit batch dimension -1, and everything works as expected.

TRTIS (and TensorRT) could document more clearly the relation between max_batch_size and EXPLICIT_BATCH. What does it mean to have max_batch_size = n, but explicit batch dimension 1? In this situation, I was able to send batches of size n, but only the first element of the batch was evaluated correctly, the other n-1 elements came back as 0.

In the end, I want to use explicit, but dynamic batch dimension, so I am ok with closing this issue. Thanks for the help.

[cong]

please use executeV2