Better support for surface layer timeseries

🚀 Feature

As discussed in this image.sc post it would be nice to provide better support for people with timeseries values of static surfaces. It would be quite easy to support an API where people could pass data in the form

vertices N x 3 (constant: doesn’t change with time)
faces M x 3 (constant: doesn’t change with time)
values L x N (each time has N values for the vertices )

and we could take care of the rest. Note that vertices could still be N x D, and values could now be D' dimensional and we’d just convert all the D'-1 dimensions into additional slider like dimensions. Should be quite clean and conform to users expectations much more strongly.

I can work on this when I get the chance, shouldn’t be too bad

[sofroniewn]

Working through an example from nilearn with surface timeseries data I encountered the following:

# NKI resting state data from nilearn
from nilearn import datasets
from nilearn import surface

nki_dataset = datasets.fetch_surf_nki_enhanced(n_subjects=1)

# Fsaverage5 surface template
fsaverage = datasets.fetch_surf_fsaverage()

# Load surface data and resting state time series from nilearn
pial_left = surface.load_surf_data(fsaverage['pial_left']) # 2-tuple
# pial_left[0].shape (10242, 3) (vertices)
# pial_left[1].shape (20480, 3) (faces)
sulc_left = surface.load_surf_data(fsaverage['sulc_left']) # shape (10242,)
timeseries = surface.load_surf_data(nki_dataset['func_left'][0]) # shape (10242, 895)

Then viewing in napari

import napari
viewer = napari.Viewer(ndisplay=3)
viewer.add_surface((pial_left[0], pial_left[1], sulc_left))
viewer.add_surface((pial_left[0], pial_left[1], timeseries[:, 0]), colormap='green') # just one timeslice

Right now creating and viewing the “timeseries compatible representation” can be done with

import numpy as np
vertices = pial_left[0]
faces = pial_left[1]
values = timeseries.T

n = len(vertices)
k = len(values)
vertices_t = np.concatenate(
    [np.concatenate((np.full((n, 1), i), vertices), axis=1) for i in range(k)],
    axis=0,
)
faces_t = np.concatenate(
    [faces + n*i for i in range(k)],
    axis=0,
)
values_t = np.concatenate([values[i] for i in range(k)])
viewer.add_surface((vertices_t, faces_t, values_t), colormap='red')

But the data is so large that viewing performance is now terrible, the refresh is too slow to be useable (though it does look cool - see below).

I think my

vertices N x D
faces M x 3
values L1 x L2 x LK x N

representation, which will result in an D+K dimensional layer isn’t too far off. We can decide though if we want

data = (vertices, faces)
values = values

or

data = (vertices, faces, values)

to be the main API. We should also do something sensible if values is not provided. Right now, it’s a bit awkward as you always have to give them in the 3-tuple. I maybe prefer the data and values separate API, it might also make this new functionality easier to understand, or make it easier to provide values for faces one day and not just vertices.

Here is the brain surface with the slider:

In general nilearn plotting is a good place to look in this context, and I am sure we could reach out to them for questions/ advice

[jni]

@sofroniewn it’s unclear right now whether this is exactly the right data model. I’d like us to look at datasets conforming to this format, e.g. fMRI datasets, and see how the data usually comes, and then try to tailor our API to that.