size error while running predict script

when I run the prediction script on a custom trained model (unlabeled paired data) I get the following error:

Traceback (most recent call last):
  File "demos/paired_MMIV/demo_predict.py", line 39, in <module>
    save_png=False,
  File "/home/charlie/3DREG-tests/DeepReg/deepreg/predict.py", line 334, in predict
    save_png=save_png,
  File "/home/charlie/3DREG-tests/DeepReg/deepreg/predict.py", line 81, in predict_on_dataset
    outputs_dict = model.predict(x=inputs_dict)
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 130, in _method_wrapper
    return method(self, *args, **kwargs)
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 1599, in predict
    tmp_batch_outputs = predict_function(iterator)
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 780, in __call__
    result = self._call(*args, **kwds)
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 814, in _call
    results = self._stateful_fn(*args, **kwds)
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 2829, in __call__
    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1848, in _filtered_call
    cancellation_manager=cancellation_manager)
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1924, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 550, in call
    ctx=ctx)
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError:  Input to reshape is a tensor with 1769472 values, but the requested shape has 3538944
         [[node DDFRegistrationModelWithoutLabel/tf_op_layer_Reshape/Reshape (defined at /home/charlie/3DREG-tests/DeepReg/deepreg/predict.py:81) ]] [Op:__inference_predict_function_6158]

Function call stack:
predict_function

the requested shape of the tensor seems to be double the one passed in input at some point and I don’t understand why, I have checked that all the shapes and the datatype of the nifti files in both training and test datasets are consistent…

any idea why this could happen?

and it occurs at the line save_png=False of the predict script but the same exact error happens with save_png=True

in that script the relevant part is:

gpu = "2,3"
gpu_allow_growth = True
predict(
    gpu=gpu,
    gpu_allow_growth=gpu_allow_growth,
    config_path=config_path,
    ckpt_path=ckpt_path,
    mode="test",
    batch_size=1,
    sample_label="all",
    log_dir=log_dir,
    save_png=False, # True is the same
)

finally, this error does not prevent the actual predictions to be output, and they look reasonable too, so I am not sure which level of the code it interferes with

[mathpluscode]

Hi again, @ciphercharly, actually, I might know the reason, you are using two GPUs, so that batch_size = 1, means 1 per GPU? Can you try using only one GPU for inference?

[ciphercharly]

thanks