Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

RuntimeError: CUDA error: invalid argument

See original GitHub issue

When I run

python test.py --cfg_file cfgs/PartA2_car.yaml --batch_size 4 --ckpt PartA2_car.pth --set MODEL.RPN.BACKBONE.NAME UNetV0 MODEL.RPN.RPN_HEAD.ARGS use_binary_dir_classifier:True

I got the following error:

2020-05-16 13:00:57,130   INFO  *************** EPOCH 2 EVALUATION *****************
eval:   0%|                                                                                                                                                                                                                                                       | 0/943 [00:00<?, ?it/s]Traceback (most recent call last):
  File "test.py", line 181, in <module>
    main()
  File "test.py", line 177, in main
    eval_single_ckpt(model, test_loader, args, eval_output_dir, logger, epoch_id)
  File "test.py", line 59, in eval_single_ckpt
    model, test_loader, epoch_id, logger, result_dir=eval_output_dir, save_to_file=args.save_to_file
  File "/home/zhanghm/Programming/PCDet/PCDet/tools/eval_utils/eval_utils.py", line 46, in eval_one_epoch
    pred_dicts, ret_dict = model(input_dict)
  File "/home/zhanghm/.virtualenvs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/zhanghm/Programming/PCDet/PCDet/pcdet/models/detectors/PartA2_net.py", line 106, in forward
    rpn_ret_dict = self.forward_rpn(**input_dict)
  File "/home/zhanghm/Programming/PCDet/PCDet/pcdet/models/detectors/PartA2_net.py", line 33, in forward_rpn
    **kwargs
  File "/home/zhanghm/.virtualenvs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/zhanghm/Programming/PCDet/PCDet/pcdet/models/rpn/rpn_unet.py", line 278, in forward
    x = self.conv_input(input_sp_tensor)
  File "/home/zhanghm/.virtualenvs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/zhanghm/.virtualenvs/pytorch/lib/python3.6/site-packages/spconv/modules.py", line 127, in forward
    input.features = module(input.features)
  File "/home/zhanghm/.virtualenvs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/zhanghm/.virtualenvs/pytorch/lib/python3.6/site-packages/torch/nn/modules/activation.py", line 99, in forward
    return F.relu(input, inplace=self.inplace)
  File "/home/zhanghm/.virtualenvs/pytorch/lib/python3.6/site-packages/torch/nn/functional.py", line 943, in relu
    result = torch.relu(input)
RuntimeError: CUDA error: invalid argument
eval:   0%|

I cann’t find any clues to solve this probelm.

Nvidia-Driver: 440.85 Cuda: 10.0

Issue Analytics

State:
Created 3 years ago
Comments:5 (1 by maintainers)

Top GitHub Comments

1reaction

youyousunnycommented, Jun 29, 2020

I have the same problem. The reason maybe is the type of CUDA. I guess the correspond type of CUDA is 9.0. But i don’t want to update the type of CUDA.

And then i have update the type of spconv. CUDA:10.0 spconv:1.1 pytorch:1.4

fix the problem. The result of inference is right in Pointpillar,Second,Pv-rcnn.

0reactions

MartinHahnercommented, Jul 11, 2020

maybe this thread can help here, too

Top Results From Across the Web

RuntimeError: CUDA error: invalid argument #1051 - GitHub

Bugs / Unexpected behaviors I try to run the code fit_textured_mesh.py(some changes have been made to make it run on my server), ...

Why do I get RuntimeError: CUDA error: invalid argument in ...

Recently I've frequently been getting RuntimeError: CUDA error: invalid argument when calling functions like torch.cholesky e.g.:

RuntimeError: CUDA error: invalid argument when running ...

For debugging consider passing CUDA_LAUNCH_BLOCKING=1. This error didn't occur when using Detectron v0.

invalid argument cuda kernel errors might be asynchronously ...

RuntimeError: CUDA error: invalid argument CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be ...

Cuda error: invalid configuration argument. (grid: 1 x 1; block

RuntimeError : Cuda error: invalid configuration argument. (grid: 1 x 1; block: 1 x 1 x 81) shape=(1,81,1,1). 455 ...