Inconsistency networks.layer.AffineTransform and transforms.Affine

Hello,

Describe the bug Applying the same affine transformation using networks.layer.AffineTransform and transforms.Affine seems to lead to different result. The layer seems to introduce an unwanted shear.

To Reproduce Steps to reproduce the behavior:

1. Download 10_3T_nody_001.nii.gz.zip
2. Run the following notebook

import monai
from monai.transforms import (
    AddChanneld,
    LoadImaged,
    ToTensord,
)
import torch as t
from copy import deepcopy

Load images and copy for comparison

path = "10_3T_nody_001.nii.gz"
add_channel = AddChanneld(keys=["image"])
loader = LoadImaged(keys=["image"])
to_tensor = ToTensord(keys=["image"])
monai_dict = {"image": path}
monai_image_layer = loader(monai_dict)
monai_image_layer = add_channel(monai_image_layer)
monai_image_layer = to_tensor(monai_image_layer)
monai_image_transform = deepcopy(monai_image_layer)

Set up example affine

translations = [0,0,0]
rotations = [t.pi/8, 0 ,0]

rotation_tensor = monai.transforms.utils.create_rotate(3, rotations,backend='torch')
translation_tensor = monai.transforms.utils.create_translate(3, translations,backend='torch')
affine =  (t.matmul(translation_tensor,rotation_tensor))

Define affine layer and transform

monai_aff_layer = monai.networks.layers.AffineTransform(mode = "bilinear",  normalized = True, align_corners= True, padding_mode = "zeros")
monai_aff_trans = monai.transforms.Affine(affine=affine, norm_coords = True, image_only = True)

Do transform using affine layer, add another dimension for batch

input_batched = monai_image_layer["image"].unsqueeze(0)

monai_image_layer["image"] = monai_aff_layer(input_batched ,t.tensor(affine).unsqueeze(0))

monai_image_layer["image"] = monai_image_layer["image"].squeeze().unsqueeze(0)

nifti_saver = monai.data.NiftiSaver(output_dir="tests", output_postfix = "layer",
                                            resample=False, padding_mode="zeros",
                                            separate_folder=False)
nifti_saver.save(monai_image_layer["image"],monai_image_layer["image_meta_dict"])

file written: tests/10_3T_nody_001_layer.nii.gz.


/var/folders/ll/btjq1lpn2msbmbn34rst5wz80000gn/T/ipykernel_64711/3194165265.py:3: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  monai_image_layer["image"] = monai_aff_layer(input_batched ,t.tensor(affine).unsqueeze(0))

Do transform using transform.Affine

monai_image_transform["image"] = monai_aff_trans(monai_image_transform["image"], padding_mode="zeros")

nifti_saver = monai.data.NiftiSaver(output_dir="tests", output_postfix = "trans",
                                            resample=False, padding_mode="zeros",
                                            separate_folder=False)
nifti_saver.save(monai_image_transform["image"], monai_image_transform["image_meta_dict"])

file written: tests/10_3T_nody_001_trans.nii.gz.

Expected behavior Unambigious behaviour of the network and transforms affine transform.

Screenshots original file: using networks.layer: using transforms.Affine

Environment

================================ Printing MONAI config…

MONAI version: 0.8.1+181.ga676e387 Numpy version: 1.20.3 Pytorch version: 1.10.2 MONAI flags: HAS_EXT = False, USE_COMPILED = False MONAI rev id: a676e3876903e5799eb3d24245872ea4f3a5b152 MONAI file: /Users/constantin/opt/anaconda3/envs/SVR_pytorch/lib/python3.9/site-packages/monai/init.py

Optional dependencies: Pytorch Ignite version: NOT INSTALLED or UNKNOWN VERSION. Nibabel version: 3.2.2 scikit-image version: 0.18.3 Pillow version: 8.4.0 Tensorboard version: 2.9.0 gdown version: NOT INSTALLED or UNKNOWN VERSION. TorchVision version: 0.11.3 tqdm version: 4.62.3 lmdb version: NOT INSTALLED or UNKNOWN VERSION. psutil version: 5.8.0 pandas version: 1.3.4 einops version: NOT INSTALLED or UNKNOWN VERSION. transformers version: NOT INSTALLED or UNKNOWN VERSION. mlflow version: NOT INSTALLED or UNKNOWN VERSION.

For details about installing the optional dependencies, please visit: https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies

================================ Printing system config…

System: Darwin Mac version: 10.16 Platform: macOS-10.16-x86_64-i386-64bit Processor: i386 Machine: x86_64 Python version: 3.9.7 Process name: python3.9 Command: [‘python’, ‘-c’, ‘import monai; monai.config.print_debug_info()’] Open files: [] Num physical CPUs: 8 Num logical CPUs: 8 Num usable CPUs: UNKNOWN for given OS CPU usage (%): [32.3, 32.3, 27.3, 25.2, 10.3, 57.9, 2.4, 1.6] CPU freq. (MHz): 2400 Load avg. in last 1, 5, 15 mins (%): [23.4, 24.4, 22.5] Disk usage (%): 54.7 Avg. sensor temp. (Celsius): UNKNOWN for given OS Total physical memory (GB): 16.0 Available memory (GB): 0.7 Used memory (GB): 1.2

================================ Printing GPU config…

Num GPUs: 0 Has CUDA: False cuDNN enabled: False

[wyli]

how one could/should just change the affine matrix. Essentially, the image is not the same anymore with different affine (it’s somewhere else in space, with other spacing etc.).

we are developing a core data structure (MetaTensor) to have the metadata fields https://github.com/Project-MONAI/MONAI/blob/f4ff8e2a07b5af29a62ef6ed14b92bd6806adc61/monai/data/meta_tensor.py#L45-L55

eventually all the monai components will be aware of the metadata (including the affine), and the metadata will be updated accordingly/automatically within the image processing workflows.

[wyli]

This will be a major step ahead - I’m looking forward to that. For me that’s for example a reason to use mainly torchio for preprocessing - as mentioned somewhere else the SpatialCrop transform is very problematic in that regard.

great feedback, thanks @Constantin-Jehn!