ERROR: infer_trtis_server.cpp:261 Triton: TritonServer response error received., triton_err_str:Internal, err_msg:PyTorch execute failure: Expected Tensor but got GenericDict

I am trying to integrate a Torchscript classifier with Deepstream 6.0. The classifier outputs a dictionary:

{'output_1': tensor([[ 3.2155, -3.0554, -0.4579,  0.7352, -4.2501,  2.6455,  0.3411]],
        grad_fn=<MmBackward>),
 'output_2': tensor([[  3.8159,   6.6151, -10.7066,  -0.4239,  -1.1774,  -5.4879,  -2.1760]],
        grad_fn=<MmBackward>)}

I tried to integrate it with Triton Server but it seems that Triton doesn’t like the dictionary. Is there a way make it work either by retrieving the dictionary or any intermediate layer? For instance, it would be great if I could retrieve the layers upon which the dictionary is computed. Then, I could recompute the final the dictionary again “by myself”.

[mfoglio]

In the end I decided to wrap the model like this:

class ModelWrapper(torch.nn.Module):

    def __init__(self, path):
        super(ModelWrapper, self).__init__()
        self.model = torch.jit.load(path, "cuda")

    def forward(self, x):
        output = self.model(x)
        return output["output_1"], output["output_2"]

[dzier]

Even with the nvinferserver, Deepstream should work with Triton, meaning that it should output Tensor Data that Triton can use. I am not sure where the dictionary datatype comes from in Deepstream. Perhaps you should ask in the Deepstream forums, since the Triton team here are not very familiar with how Deepstream works.