FileNotFoundError: File not found: "path" when data from subject dataset is fetched by dataloader

Is there an existing issue for this?

• I have searched the existing issues

Problem summary

Hi, I’m having a very strange error. When I’m fethcing an object with dataloader, it returns this error:

FileNotFoundError: File not found: “path”

I debugged and found out that “somehow” an object of ScalarImage is being created by the collate function of the dataloader which receives a dictonary of the subject as a “path” in the constructor of the ScalarImage class.

I couldn’t find any reason why and how it is hapenning.

One thing I did manage to figure out: it happens with the nightly build of pytorch.

When I’m using Stable (1.10.2) of PyTorch, I do not encounter this error. When I’m using the nightly build (currently, 1.11.0.dev20220130), then I encounter this error.

I have tested other subject classes (non-TorchIO) with the nightly builds, and they are working fine.

Here’s the complete project where I got this error: https://github.com/soumickmj/NCC1701 The code snippet for reproduction is given below.

Do you have any idea what might be the trouble?

Thanks!

Code for reproduction

file = "dummy/IXI019-Guys-0702-T1.nii.gz"

sub = tio.Subject(inp=tio.ScalarImage(file),
                gt=tio.ScalarImage(file),
                filename=os.path.basename(file),
                processed=False
)

subjects = 2 * [sub]
dataset = tio.SubjectsDataset(subjects)
loader = torch.utils.data.DataLoader(dataset , batch_size=16)

a = next(iter(loader))

Actual outcome

one subject element

Error messages

Traceback (most recent call last):
  File "/project/schatter/bin/anaconda3/envs/torchMRIBetapy92/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/project/schatter/bin/anaconda3/envs/torchMRIBetapy92/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/schatter/.vscode-server-insiders/extensions/ms-python.python-2021.12.1559732655/pythonFiles/lib/python/debugpy/__main__.py", line 45, in <module>
    cli.main()
  File "/home/schatter/.vscode-server-insiders/extensions/ms-python.python-2021.12.1559732655/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 444, in main
    run()
  File "/home/schatter/.vscode-server-insiders/extensions/ms-python.python-2021.12.1559732655/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 285, in run_file
    runpy.run_path(target_as_str, run_name=compat.force_str("__main__"))
  File "/project/schatter/bin/anaconda3/envs/torchMRIBetapy92/lib/python3.9/runpy.py", line 268, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/project/schatter/bin/anaconda3/envs/torchMRIBetapy92/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/project/schatter/bin/anaconda3/envs/torchMRIBetapy92/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/data/project/SoumickPavan/NCC1701Code/a.py", line 17, in <module>
    a = next(iter(loader))
  File "/project/schatter/bin/anaconda3/envs/torchMRIBetapy92/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 530, in __next__
    data = self._next_data()
  File "/project/schatter/bin/anaconda3/envs/torchMRIBetapy92/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 570, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/project/schatter/bin/anaconda3/envs/torchMRIBetapy92/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
    return self.collate_fn(data)
  File "/project/schatter/bin/anaconda3/envs/torchMRIBetapy92/lib/python3.9/site-packages/torch/utils/data/_utils/collate.py", line 157, in default_collate
    return elem_type({key: default_collate([d[key] for d in batch]) for key in elem})
  File "/project/schatter/bin/anaconda3/envs/torchMRIBetapy92/lib/python3.9/site-packages/torch/utils/data/_utils/collate.py", line 157, in <dictcomp>
    return elem_type({key: default_collate([d[key] for d in batch]) for key in elem})
  File "/project/schatter/bin/anaconda3/envs/torchMRIBetapy92/lib/python3.9/site-packages/torch/utils/data/_utils/collate.py", line 157, in default_collate
    return elem_type({key: default_collate([d[key] for d in batch]) for key in elem})
  File "/project/schatter/bin/anaconda3/envs/torchMRIBetapy92/lib/python3.9/site-packages/torchio/data/image.py", line 746, in __init__
    super().__init__(*args, **kwargs)
  File "/project/schatter/bin/anaconda3/envs/torchMRIBetapy92/lib/python3.9/site-packages/torchio/data/image.py", line 161, in __init__
    self.path = self._parse_path(path)
  File "/project/schatter/bin/anaconda3/envs/torchMRIBetapy92/lib/python3.9/site-packages/torchio/data/image.py", line 428, in _parse_path
    return [self._parse_single_path(p) for p in path]
  File "/project/schatter/bin/anaconda3/envs/torchMRIBetapy92/lib/python3.9/site-packages/torchio/data/image.py", line 428, in <listcomp>
    return [self._parse_single_path(p) for p in path]
  File "/project/schatter/bin/anaconda3/envs/torchMRIBetapy92/lib/python3.9/site-packages/torchio/data/image.py", line 418, in _parse_single_path
    raise FileNotFoundError(f'File not found: "{path}"')
FileNotFoundError: File not found: "path"

Expected outcome

one subject element

System info

Platform:   Linux-5.4.0-81-generic-x86_64-with-glibc2.31
TorchIO:    0.18.73
PyTorch:    1.11.0.dev20220130
SimpleITK:  2.1.1 (ITK 5.2)
NumPy:      1.21.2
Python:     3.9.7 (default, Sep 16 2021, 13:09:58) 
[GCC 7.5.0]

[snipdome]

Hi @ll, Thanks for reporting a detailed issue report, Soumick! As you know, I submitted a fix the error, which is currently in pull-request

What is happening: The last called torch function default_collate() has been changed (from torch>=1.11.0) in order to support a bigger variety of user-cases. By the way it has been modified, it calls the init() of the input batch-type (in our case, we are talking about Subject) and, if it raises a TypeError (standard exception for wrong type of arguments as input), then the pytorch code could continue - basically running the same code that we have been running with previous versions of pytorch. The latter branch of the code, does not call init(), which I believe it is indeed what we want, since the Subject class goes out of scope after a batch has been prepared. Right now, there are two things that are not ideal:

• class Image does not recognize on the spot that it has been called with the wrong type of data (path as dictionary)
• class Subject raise a ValueError instead of TypeError, when it is called with no supplied Images

With the proposed fix, the kind of raised exception changes, as well as an additional check has been added in Image.

[ryancinsight]

same issue on windows, just installed the new pytorch dev too, and replicated back 0.18.0