Clarifying the interface between the image layer and data

🚀 Feature

(This is a follow-up to discussions that occurred offline, and on zulip)

Currently there is a lot of logic in the image layer for handling different types of data, e.g. arrays that represent grayscale data (the default), arrays that represent RGB data, and image pyramids (lists of arrays). Accordingly, the image layer has rgb and is_pyramid properties, which the image layer uses when figuring out how get images from the user-supplied data, e.g. in Image._set_view_slice.
There are several standing issues that might further complicate the data wrangling in the image layer, namely adding functionality for lazy loading of planar views (https://github.com/napari/napari/issues/318) and adding support for a physical coordinate model (https://github.com/napari/napari/issues/763). I think these issues might be simpler to resolve if there’s a cleaner separation of concerns between the image layer and the source of the data that the image layer is displaying. I know @jni’s blood pressure shoots up when he hears the O word, but I want to use this issue to explore the advantages and disadvantages of making an object (tentatively called a datasource) to take a lot of data logic out of the image layer class.

Schematically, I’m a imagining what life would be like if the image layer had a datasource object which could act like a locus for data properties and access methods, e.g.

• dimensional information / units (e.g., how the data grid maps to world coordinates)
• min and max values of the data (which are not the same as the layer’s contrast limits)
• whether the data has changed (maybe important for streaming camera data)
• whether the data is RGB or not
• a method that takes some spatial parameters and returns a view of the data corresponding to those parameters. By making this a method call and not just using array_like.__getitem__, it becomes a lot easier to pass in arrays of (eventually) floats for resampling data off-grid.

I’m still not super sure what exactly this object should look like, or what it should do. But I think adding a little bit of abstraction in this area could make solving some current issues a lot easier.

To help with https://github.com/napari/napari/issues/763, a datasource object could totally encapsulate the mapping from the data grid to a grid in the viewer space. That information (e.g., an affine matrix and a transform method) needs to live somewhere; a datasource-like object seems like a good fit for it.

To help with https://github.com/napari/napari/issues/318, a datasource object, when generating views of a lazy array like a dask array, could return to the layer an array of NaNs and schedule a reader thread that progressively fills in missing values of that array. Especially for testing & development, it seems more natural for this tiled loading functionality to be encapsulated in an object with a relatively restricted interface, as opposed to bolting it on to the image layer, which has lots of visualization-specific concerns (colormaps, etc).

Unfortunately I don’t have any code yet, these are just ideas… I’m spending a lot of time staring at code, thinking about what structure is right.

[d-v-b]

I wonder how much of this stuff we could get for free by using xarray. “rgbness” of an xarray can be represented by assigning the name “rgb” to the axis corresponding to the color channel; the coords property of an xarray naturally describes the position of the data in physical space. It should also be simple to use xarrays for making lazy pyramids, since they have a coarsen method… I’ll see what this looks like in code form.

[royerloic]

Just noticed that I did not weigh in on @sofroniewn question on ‘NaNs’: “… how having that array of NaNs is going to interact with our vispy code”. For sure this won’t work – NaNs are not interpreted by OpenGL – unless we do something just before we sent the data to the driver. The right approach as suggested is to use a tilling approach. I think this is clear by now, but just wanted to have stated clearly here since the question was asked 😃