Runtime error in torchvision nms in Linux. Windows works fine

Hi,

I’m getting an runtime error when running torchvision\ops\boxes.nms. torchvision 0.4.0 pytorch 1.2.0 (GPU)

RuntimeError: Trying to create tensor with negative dimension -532064992: [-532064992] (check_size_nonnegative at /pytorch/aten/src/ATen/native/TensorFactories.h:64)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x33 (0x7fd8e1d79273 in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: at::native::empty_cuda(c10::ArrayRef, c10::TensorOptions const&, c10::optionalc10::MemoryFormat) + 0xb76 (0x7fd7d85e8fb6 in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch.so)
frame #2: + 0x3f7da58 (0x7fd7d6f7fa58 in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch.so)
frame #3: torch::autograd::VariableType::empty(c10::ArrayRef, c10::TensorOptions const&, c10::optionalc10::MemoryFormat) + 0x3fa (0x7fd7d69fa75a in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch.so)
frame #4: + 0x7f661 (0x7fd87b3fd661 in /opt/conda/lib/python3.6/site-packages/torchvision/_C.cpython-36m-x86_64-linux-gnu.so)
frame #5: nms_cuda(at::Tensor const&, at::Tensor const&, float) + 0x430 (0x7fd87b3fe042 in /opt/conda/lib/python3.6/site-packages/torchvision/_C.cpython-36m-x86_64-linux-gnu.so)
frame #6: nms(at::Tensor const&, at::Tensor const&, float) + 0x172 (0x7fd87b3c1cf9 in /opt/conda/lib/python3.6/site-packages/torchvision/_C.cpython-36m-x86_64-linux-gnu.so)
frame #7: + 0x65115 (0x7fd87b3e3115 in /opt/conda/lib/python3.6/site-packages/torchvision/_C.cpython-36m-x86_64-linux-gnu.so)
frame #8: + 0x62304 (0x7fd87b3e0304 in /opt/conda/lib/python3.6/site-packages/torchvision/_C.cpython-36m-x86_64-linux-gnu.so)
frame #9: + 0x5dc45 (0x7fd87b3dbc45 in /opt/conda/lib/python3.6/site-packages/torchvision/_C.cpython-36m-x86_64-linux-gnu.so)
frame #10: + 0x5ded2 (0x7fd87b3dbed2 in /opt/conda/lib/python3.6/site-packages/torchvision/_C.cpython-36m-x86_64-linux-gnu.so)
frame #11: + 0x4f2e7 (0x7fd87b3cd2e7 in /opt/conda/lib/python3.6/site-packages/torchvision/_C.cpython-36m-x86_64-linux-gnu.so)

To Reproduce code:

nms(transformed_anchors, scores, iou_threshold = 0.7)
transformed_anchors is [ 490698, 4]
scores is [ 490698]

transformed_anchors .min () = 0
transformed_anchors .max () = 2560

Environment

[pip] msgpack-numpy==0.4.3.2
[pip] numpy==1.16.4
[pip] torch==1.2.0
[pip] torchtext==0.4.0
[pip] torchvision==0.4.0
[conda] magma-cuda100 2.1.0 5 local
[conda] mkl 2019.1 144
[conda] mkl-include 2019.1 144
[conda] torch 1.2.0 pypi_0 pypi
[conda] torchtext 0.4.0 pypi_0 pypi
[conda] torchvision 0.4.0 pypi_0 pypi

Additional context same code on Windows runs as expected

same issue opened in pytorch/pytorch

[fmassa]

Hi @senarvi,

I agree that this error message is not great. We should at least make the error message more meaningful by replacing the int with int64_t.

[senarvi]

Even if the implementation doesn’t scale up to larger sizes, it would be great to get a better error message.