No ray-voxel intersections occurring: 'NoneType' object has no attribute 'new_ones'

Dataset to recreate

Describe the bug I’ve been able to get NSVF to successfully run locally on the supplied datasets, but I’m running into errors when using my own custom dataset. I converted my dataset to match the format described in the README and set the bounding box dimensions to the minima and maxima of the translation coordinates among all extrinsic camera matrices (pose matrices). I calculated the voxel size according to the volume of these boundaries.

Running train.py with the supplied arguments results in the following error:

File "train.py", line 20, in <module>
    cli_main()
  File "/home/ubuntu/CSM/NSVF/fairnr_cli/train.py", line 373, in cli_main
    main(args)
  File "/home/ubuntu/CSM/NSVF/fairnr_cli/train.py", line 104, in main
    should_end_training = train(args, trainer, task, epoch_itr)
  File "/usr/lib/python3.7/contextlib.py", line 74, in inner
    return func(*args, **kwds)
  File "/home/ubuntu/CSM/NSVF/fairnr_cli/train.py", line 181, in train
    log_output = trainer.train_step(samples)
  File "/usr/lib/python3.7/contextlib.py", line 74, in inner
    return func(*args, **kwds)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/fairseq/trainer.py", line 431, in train_step
    ignore_grad=is_dummy_batch,
  File "/home/ubuntu/CSM/NSVF/fairnr/tasks/neural_rendering.py", line 300, in train_step
    return super().train_step(sample, model, criterion, optimizer, update_num, ignore_grad)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/fairseq/tasks/fairseq_task.py", line 351, in train_step
    loss, sample_size, logging_output = criterion(model, sample)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/CSM/NSVF/fairnr/criterions/rendering_loss.py", line 42, in forward
    net_output = model(**sample)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/CSM/NSVF/fairnr/models/fairnr_model.py", line 77, in forward
    results = self._forward(ray_start, ray_dir, **kwargs)
  File "/home/ubuntu/CSM/NSVF/fairnr/models/nsvf.py", line 91, in _forward
    all_results['missed'] = fill_in((fullsize, ), hits, all_results['missed'], 1.0).view(S, V, P)
  File "/home/ubuntu/CSM/NSVF/fairnr/data/geometry.py", line 306, in fill_in
    output = input.new_ones(*shape) * initial
AttributeError: 'NoneType' object has no attribute 'new_ones'

I investigated further and determined that this means no rays are intersecting with the voxels during training. I assume that this is due to an issue with my bounding box setup. To recreate this, here is my dataset.

Is there an issue with the way I’m using my camera pose matrices to set my bounding box dimensions? Are there other steps I could take to ensure that at least some intersections are occurring between the rays and voxels?

[dukeeagle]

Thanks for the reply! The issue with backgrounds makes sense. All I want captured is the chair in the center of the scene, so I don’t care if the background gets lost. Is there a method you’d recommend for setting the bounding box to capture just the chair? Or is there a method you’d recommend for automatically removing the background from each RGB image?

Additionally, I noticed that Figure 7 of the paper runs NSVF on a Scannet scene that seems to be unbounded an include a background. How did you and the team achieve this given how non-white backgrounds are not allowed?

I also saw that the background was removed from this truck in the Tanks&Tamples dataset in *Figure 11 of the paper. Did you do this manually or is there a special technique you used to achieve a white background?

Thanks for all the help on this! It’s been super informative already.

[MultiPath]

Sorry for late reply! I will take a look at your dataset tomorrow and see how to debug it.