ValueError: when running VarNet

I get the following ValueError when I attempt to run the VarNet. Any idea why? I am using the NYU multi-coil knee dataset but just limited (10 training h5py files). I have in my environment pytorch-lightning 0.6.0 and torch 1.3.1 with torchvision 0.4.2.

This is what I am using to train:

python models/varnet/varnet.py --resolution 320 --mode train --challenge multicoil --exp var_net --mask-type random --data-path /media/iva19/multicoil_train/

and that’s the error:

INFO:root:gpu available: True, used: True
INFO:root:VISIBLE GPUS: 0
Traceback (most recent call last):
  File "models/varnet/varnet.py", line 374, in <module>
    main()
  File "models/varnet/varnet.py", line 371, in main
    run(args)
  File "models/varnet/varnet.py", line 342, in run
    trainer.fit(model)
  File "/home/iva19/usr/local/miniconda3/envs/fastMRI/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 687, in fit
    mp.spawn(self.ddp_train, nprocs=self.num_gpus, args=(model,))
  File "/home/iva19/usr/local/miniconda3/envs/fastMRI/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn
    while not spawn_context.join():
  File "/home/iva19/usr/local/miniconda3/envs/fastMRI/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 118, in join
    raise Exception(msg)
Exception: 

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/home/iva19/usr/local/miniconda3/envs/fastMRI/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
  File "/home/iva19/usr/local/miniconda3/envs/fastMRI/lib/python3.6/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 331, in ddp_train
    self.run_pretrain_routine(model)
  File "/home/iva19/usr/local/miniconda3/envs/fastMRI/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 757, in run_pretrain_routine
    self.logger.log_hyperparams(ref_model.hparams)
  File "/home/iva19/usr/local/miniconda3/envs/fastMRI/lib/python3.6/site-packages/pytorch_lightning/logging/base.py", line 14, in wrapped_fn
    fn(self, *args, **kwargs)
  File "/home/iva19/usr/local/miniconda3/envs/fastMRI/lib/python3.6/site-packages/pytorch_lightning/logging/tensorboard.py", line 88, in log_hyperparams
    self.experiment.add_hparams(hparam_dict=params, metric_dict={})
  File "/home/iva19/usr/local/miniconda3/envs/fastMRI/lib/python3.6/site-packages/torch/utils/tensorboard/writer.py", line 292, in add_hparams
    exp, ssi, sei = hparams(hparam_dict, metric_dict)
  File "/home/iva19/usr/local/miniconda3/envs/fastMRI/lib/python3.6/site-packages/torch/utils/tensorboard/summary.py", line 156, in hparams
    raise ValueError('value should be one of int, float, str, bool, or torch.Tensor')
ValueError: value should be one of int, float, str, bool, or torch.Tensor

[mmuckley]

I saw that as well. This model is pretty heavy on memory, also on my 16 GB GPU. Perhaps they prototyped it on a 32 GB GPU.

I was able to get past this error by decreasing the size of the model - e.g., --num-cascades 4.

[mmuckley]

Great. I’m going to be going through the repository soon and trying to clean up a few things, including requirements.txt.