Add support for 4D images

🚀 Feature

Add support to read, write, sample and transform 4D images.

Motivation

It would be convenient to support 4D images out of the box, as tensors of shape (C, S1, S2, S3) where C is the number of channels and SX are spatial dimensions (S1 = 1 for 2D images).

Examples of what the 4th dimension might encode

Gradient directions

• Diffusion MRI

Time

• Functional MRI
• Ultrasound sequence

Modalities or MRI sequences

• T1, CT, US
• T1, T2, PD

EM frequency bands

• RGB
• Multispectral

Labels

• Discrete: one-hot encoded labels
• Continuous: fractional labelmap aka fuzzy volumes aka tissue probability maps

Pitch

Support of 4D images in:

• torchio.Image
• torchio.io
• Samplers
• Intensity transforms
• Spatial transforms
• Docs

Alternatives

Pass the channels independently in different instances of Image.

Additional context

Considerations

torchio.io

Possible image shapes when reading

How do we know if 2D or 3D when there are 3 dimensions in the read data? Lookup table of file extensions?

2D

• (height, width)
• (height, width, channels): how do we know whether the 3rd dimension is depth or channels? Easy to infer if it’s 3 as probably RGB, harder otherwise (e.g. multispectral images).

3D

• (height, width, depth): how do we know whether the 3rd dimension is depth or channels?
• (height, width, depth, channels): is this typical for dMRI and fMRI? What about 4D ultrasound?

Transforms

Consider whether to apply the same or different transformations to each channel of each image, as pointed out by @romainVala in https://github.com/fepegar/torchio/issues/238#issuecomment-661708748.

Related issues

• #100: Consider multi-channel output in the GridAggregator
• #213: Support 4D tensors as torchio.Image
• #215: Transform torchio.Subject - mask with multiple masks and different shapes than image
• #228: Passing mean and std to normalization transform
• #234: Inconsistent shapes for diffusion MRI
• #236: How to handle multi-modal images?
• #237: How to handle multi-class segmentation?

[meghbhalerao]

Thanks for the clarification @fepegar!

[romainVala]

Great

There is no obvious difficulties for read write and sample since it is just adding multi-channel …

Here are just a few thought for what to do with transform

For spatial transform : one would like to have the same transfo apply to each channel. Either a for loop on the channel, on directly apply on 4D, (which is strangely slower for resample as shown in #213 )

Although in the case of diffusion one may be interested, to produce different elastic transform (as there are small non linear deformation between diffusion volume) … so may be the same option as for intensity ?

For intensity transform: we should have the choice (with a user define parameter) between applying the same transform for each channel or a different one

As an example for Motion, with diffusion or fMRI data one expect to have a different motion for each volume, but with a multi echo sequence, we expect the same motion for each volume

It seem logical that for random_skipke we use a different one, but for random_bias the same for each channel … but I suggest to let the user decide

this may be quite a lot of code change in each transform …