output from trt model is different from pytorch

Hi, I see a big difference in output values for trt model after conversion vs original pytorch model. The network that I’m converting is not complex and it does not give any error when doing torch2trt conversion. The network is linked here

I do the conversion using this line pnet_trt = torch2trt(pnet, [1,3,109,193], fp16_mode=False)

and use these lines to measure the difference, the difference is around 0.7 print(output.flatten()[0:10]) print(output_trt.flatten()[0:10]) print('max error: %f' % float(torch.max(torch.abs(output - output_trt))))

I am trying on Jetson Nano with Jetpack 4.4 CUDA 10.2, CuDNN 8.0, TensorRT 7.1.0, PyTorch 1.5.0

thank you

[F-A]

@yptheangel TensorRT operates directly on the data buffer (cf links I posted ealier: https://github.com/NVIDIA-AI-IOT/torch2trt/issues/220#issuecomment-569949961). This means that it works directly on the memory and assumes that the tensor data is contiguously stored. If it is not the tensorRT model will still access the data as the place and might therefore get random data as input.

If also found out that in some case the model needs to be in evaluation model before the torch2rt conversion, otherwise there is some slights changes in the tensortRT outputs.

[Secondgrade]

@Secondgrade for MTCNN, The RNet and ONet, I explicitly set that the TRT network to have a fixed batch size, I set 100. Earlier on, TRT could not have dynamic batch size network. but RNet and ONet is not the slow ones, the PNet takes up most of the compute time.

I have not figure out a very good way to convert PNet to TRT form. As PNet handles multiple scales, I have tried converting the PNet into multiple tiny networks, but there was no speedup.

You can try stitching the input images at multiple scales and then do a 1 time inference, if it works then we can convert into to TRT and check out the inference time difference. check out this blog

OK, thanks for your reply.