RandAffined does not work for differently sized images

Describe the bug

When the images in a pipeline entry have different sizes, then the RandAffined does not work properly. The resulting images no longer match.

To Reproduce

Steps to reproduce the behavior:

1. Load these files: example_files.zip. You can view them with Slicer or MRIcroGL (itk-snap seems to have a bug which prevents it to load these files).
2. Execute this pipeline:

from pathlib import Path

import numpy as np
from monai.transforms.compose import Compose
from monai.transforms.io.dictionary import LoadImaged, SaveImaged
from monai.transforms.spatial.dictionary import RandAffined
from monai.transforms.utility.dictionary import (EnsureChannelFirstd, ToTensord)
from monai.utils.enums import GridSampleMode

path = Path('scratch/rand_affine')

T1_KEY = 't1'
LABEL_KEY = 'labels'
FILE_KEYS = [T1_KEY, LABEL_KEY]
SAMPLE_MODES = [GridSampleMode.BILINEAR, GridSampleMode.NEAREST]

data = {
    T1_KEY: path / 't1.nii.gz',
    LABEL_KEY: path / 'labels.nii.gz',
}

process = Compose([
    LoadImaged(FILE_KEYS),
    ToTensord(FILE_KEYS),
    EnsureChannelFirstd(FILE_KEYS),
    RandAffined(
        FILE_KEYS,
        rotate_range=5.0 * np.pi / 180.0,  # type: ignore
        prob=1.0,
        mode=SAMPLE_MODES,
        allow_missing_keys=True,
    ),
    SaveImaged(
        FILE_KEYS,
        output_dir=path,
        output_postfix=f"out",
        resample=False,
        output_dtype=np.float32,
        separate_folder=False,
        print_log=False,
    )
])

process([data])

Expected behavior The same random affine transformation is applied to all images. After the transform the labels still match the image.

Screenshots

Given the following image and label:

When applying RandAffined, the image and labels no longer match:

Additional context

We are using torchio’s RandomAffine transform instead.

This issue was previously closed (https://github.com/Project-MONAI/MONAI/issues/3316) and moved to a discussion (https://github.com/Project-MONAI/MONAI/discussions/3319) due to a bug in itk-snap. Since this does not have anything to do with this bug report, I’m reopening it.

[Spenhouet]

Hi @Nic-Ma,

good to hear. I believe this is a worth while undertaking. I think the current way the transforms and meta_dict are handled via extended key string is not great. If this could be unified and simplified this would help immensely for this. If this information would always be at the same place or always part of every image, this would remove quite some boilerplate.

Best regards, Sebastian

Edit: Basically, adding a form of Image object which then always requires certain information like an affine or spacing.

[Nic-Ma]

Hi @Spenhouet ,

Thanks very much for your great feedback and suggestions. We discussed these medical-image processing related feature requests last week, the main missing part in current codebase is how to maintain the metadata during transforms. It would be a very big task for us to update metadata in every transform, but we already started to investigate the details.

Thanks.