Allow batch size 0 in ensembles

This is related to #443 and #509

The use case is as follows: I want to take the output of an object detector A, feed it to a custom backend B which creates crops from the original image and feed the crops to a further model C.

This works wonderfully using TRTIS’ ensemble scheduling. With one exception: The output of the custom backend has shape [N,3,h,w] where N is the number of detected bounding boxes. If there is no detection, I would like to write an “empty” tensor as output, e.g. a tensor of shape [0,3,h,w], skip evaluation of model C and set the outputs of C also to be of batch size 0. In the custom backend I request such a buffer (of size 0) from TRTIS. But on inference TRTIS complains that the input to model C is too small.

My workaround is to always write some crop and just ignore the output of C if no bounding boxes were found. But this is clearly not optimal.

Is a fix for this possible/in the pipeline?

Edit: I should have said that model C is a TensorRT plan with optimization profile with minimal input shape [1,3,h,w]. TensorRT understandably does not allow minimal input shape [0,3,h,w].

[NotAnyMike]

Has something changed lately on this topic? I have faced this issue several times and is actually stopping us from fully moving to triton

[deadeyegoodwin]

That would be very specific behavior for TRTIS to implement… and then we would have to construct some appropriate outputs even-though the model itself didn’t run. It is unlikely that this is something we would put in the ensembler. I think your current workaround is reasonable given the TensorRT limitation of not allowing zero-sized dimensions.