elastix resampled image from transformix and ItkTransform resampled image produce different results for same transformations

Hi all, specifically @ViktorvdValk @N-Dekker

I’ve been monitoring a bit the GitHub issues and upcoming PRs for ITKElastix and elastix itself and thought it was timely to bring this issue up considering the direction things are going for interoperability between elastix and ITK transforms. I wasn’t sure exactly where to file this so I will put it here and have tagged who I think would be interested.

Related issues: #79 elastix: https://github.com/SuperElastix/elastix/pull/387

Background

I am registering some 2D microscopy images and having great results with elastix, however, I’d like to use a bit more of ITK for managing point set transformations and composing long and complex sequences of transforms. However, in my initial tests I’ve discovered that the resampling in ITK and elastix can produce different images with spatial shifts that could affect point set and image transformations.

Reproducible example & data

Here is a full example with data (data link: https://drive.google.com/file/d/1-bUfi69YmBVGFPzX4XRTPPCkIja9Qur8/view?usp=sharing, sorry it is microscopy so rather large 1.1 GB, includes both the moving image, and ITK and elastix transformed images) using SimpleITK but I think it would apply to ITK or elastix implementation one uses. Apologies for the length, I wanted everything to be as clear as possible.

import numpy as np
import SimpleITK as sitk
import napari


def elx_to_itk_euler2d(transform_data):
    t_params = np.asarray(transform_data["TransformParameters"]).astype(np.float)
    t_center = np.asarray(transform_data["CenterOfRotationPoint"]).astype(np.float)

    theta = t_params[0]
    c, s = np.cos(theta), np.sin(theta)
    rotmat = np.array(((c, -s), (s, c)))

    affine = sitk.AffineTransform(2)
    affine.SetTranslation(t_params[-2:].tolist())
    affine.SetMatrix(rotmat.flatten().tolist())
    affine.SetCenter(t_center.tolist())
    return affine


def elx_to_itk_affine2d(transform_data):
    t_params = np.asarray(transform_data["TransformParameters"]).astype(np.float)
    t_center = np.asarray(transform_data["CenterOfRotationPoint"]).astype(np.float)

    affine = sitk.AffineTransform(2)
    affine.SetTranslation(t_params[-2:].tolist())
    affine.SetMatrix(t_params[:4].flatten().tolist())
    affine.SetCenter(t_center.tolist())
    return affine


# read elastix parameter files
euler_pmap = sitk.ReadParameterFile("./example-data/TransformParameters.0.txt")
affine_pmap = sitk.ReadParameterFile("./example-data/TransformParameters.1.txt")

# read moving image and set spacing
moving_im = sitk.ReadImage("./example-data/moving_image.tif")
moving_im.SetSpacing((0.65, 0.65))

# set up fixed image from last transform in sequence
size_out = np.asarray(affine_pmap["Size"]).astype(np.int)
spacing_out = np.asarray(affine_pmap["Spacing"]).astype(np.float)
# direction and origin are default
fixed_im = sitk.Image(size_out.tolist(), sitk.sitkUInt16)
fixed_im.SetSpacing(spacing_out.tolist())

# convert elastix transform parameter maps to ITK representations
itk_euler = elx_to_itk_euler2d(euler_pmap)
itk_affine = elx_to_itk_affine2d(affine_pmap)

# compose sequential transforms
compos_tform = sitk.Transform(2, sitk.sitkComposite)
compos_tform.AddTransform(itk_euler)
compos_tform.AddTransform(itk_affine)

# apply transform
itk_resampled_im = sitk.Resample(
    moving_im, fixed_im, compos_tform, sitk.sitkNearestNeighbor, 0
)

# load elastix resampled image
elx_resampled_im = sitk.ReadImage("./example-data/elx_resampled_image.tif")

# optionally read the ITK resampled image from disk rather than computing it
# itk_resampled_im = sitk.ReadImage("./example-data/itk_resampled_image.tif")

# visualize with napari
with napari.gui_qt():
    viewer = napari.Viewer()
    viewer.add_image(
        sitk.GetArrayFromImage(itk_resampled_im),
        name="itk resampled",
        contrast_limits=[0, 5000],
        colormap="red",
        blending="additive"

    )
    viewer.add_image(
        sitk.GetArrayFromImage(elx_resampled_im),
        name="elx resampled",
        contrast_limits=[0, 5000],
        colormap="green",
        blending="additive"
    )

Results

Here are the results with red-green (elastix, itk) additive color blending of the different resamplings: low-res is ok

high-res shows some issues, a small positive shift of itk image

I’ve also looking at corresponding areas between the original moving image and the different resamplings and elastix’s resampled image has higher fidelity to the original pixel intensities and shapes and is a closer alignment with the fixed image.

Not sure what is the cause of this. Other things I’ve explored producing the same result:

• downsampling of the image
• precision of numbers in TransformParameters
• sequence transformation rather than ITK composition transform, i.e., transform with euler -> interpolate -> affine -> interpolate

Sorry for the long issue… hope it is clear.

[NHPatterson]

@mstaring Reverse the order seems to have worked! I’m really surprised I didn’t try this, it seems so obvious. It was also important to have the higher precision numbers as I have tested on an older version of ITKElastix as well.

@thewtex This will affect the resampling notebook you have uploaded. Another note there: the default interpolator is Linear (I believe) for itk resampling whereas the elastix method is a “FinalBSplineInterpolator” with order = 3.

# switch ordering
composite_transform = itk.CompositeTransform[itk.D, dimension].New()
composite_transform.AddTransform(affine_transform)
composite_transform.AddTransform(euler_transform)

# set interpolation
interpolator = itk.BSplineInterpolateImageFunction.New(moving_image)
interpolator.SetSplineOrder(3)

# add interpolation kwarg
result_image_itk = itk.resample_image_filter(moving_image,
                                             transform=composite_transform,
                                             use_reference_image=True,
                                             reference_image=fixed_image,
                                             interpolator=interpolator)

Although I will say for the example notebook the results are not exactly equal for BSpline interpolation whereas the arrays are exactly equal for nearest neighbor and linear.

[NHPatterson]

I did update to the last version of everything and now the parameters are serialized to a more precise number, but the problem persists exactly as before.

I think the issue is in the compositing of transforms. In the registration I perform a euler transform then follow with an affine transform. I tried running each separately result in 1 transform for each and in both the cases the resampling by ITK and elastix are perfectly matched.

Affine example

I also tried an iterative approach. Running the two alignments separately (register, take registered image output, register this output again to fixed, see code below) and then composing the transforms through ITK.

parameter_object_euler = itk.ParameterObject.New()
parameter_object_euler.AddParameterMap(euler_params)
result_image_euler, result_transform_parameters_euler = itk.elastix_registration_method(
    fixed_im, moving_im,
    parameter_object=parameter_object_euler,
    log_to_console=True)

parameter_object_affine = itk.ParameterObject.New()
parameter_object_affine.AddParameterMap(affine_params)
result_image_affine, result_transform_parameters_affine = itk.elastix_registration_method(
    fixed_im, result_image_euler,
    parameter_object=parameter_object_affine,
    log_to_console=True)

The error is now much smaller but there is still some error present but it’s much smaller.

I also tried concatenating the elastix transforms through transformix like so:

transformix_object = itk.TransformixFilter.New(moving_im)
tfx_params = itk.ParameterObject.New()
tfx_params.AddParameterMap(result_transform_parameters_euler.GetParameterMap(0))
tfx_params.AddParameterMap(result_transform_parameters_affine.GetParameterMap(0))
transformix_object.SetTransformParameterObject(tfx_params)

transformix_object.UpdateLargestPossibleRegion()
result_image_transformix = transformix_object.GetOutput()

but this image is significantly wrong result which seems like an elastix error, it works when one does transformix twice.