Datashader support

The new datashader support in HoloViews is great – we can now easily choose between a “normal” HoloViews plot, a decimated one, or an aggregated one, making it simple to work with large or small datasets. But there are still a few wrinkles to iron out:

Automatic bin array sizing

The height and width of the datashader bin array must currently be set explicitly, but the plot size in pixels is available from the underlying Bokeh plot, and so we should be able to attach some streams that provide that size to datashader for its bin array dimensions instead. It will always be important for users to be able to override such automatic values, but having them be automatic would be vastly more convenient in most cases.

Unifying shading and colormapping

There are now two completely separate mechanisms for “shading”, i.e., rendering a data array stored in an hv.Image object into a set of RGBA pixels for display. The “backend” mechanism is called by the display hook for hv.Image objects, to provide “rich display” of the object in Jupyter. This mechanism provides a colormap to the backend (Matplotlib or Bokeh) and requests that the backend render the image. Presumably the backend mechanism allows Bokeh to provide accurate hover information, as it has direct access to the underlying data values. The backend mechanism is also fully automatic, from the user perspective, being called behind the scenes whenever such an object needs to be displayed.

The other mechanism is provided by hv.operation.datashader.shade(), which explicitly turns an hv.Image object into an hv.RGB object, performing the colormapping from within Python using specialized functions in Datashader. The shade() mechanism has functionality not present in Bokeh or Matplotlib’s colormapping, namely doing histogram equalization, using the alpha channel to provide a mask, and allowing arbitrary data normalization functions if needed. However, it currently must be invoked explicitly, using shade() to create a different HoloViews object type that no longer has information about the original data values, and is thus a “dead end” just for visualization, of the type that we try to avoid in HoloViews. It would be nice to put this functionality into the plot options instead so that it can be invoked automatically on rich display, to avoid such dead ends and make the user experience more straightforward.

Shading categorical aggregates

Categorical aggregates currently end up as an NdOverlay of Images, which can’t normally be viewed directly as the topmost image obscures all the rest. Thus these aggregates have an additional wrinkle similar to the issue for shading above – they must first be converted explicitly into something visible, rather than allowing the user to use the same representation for working with data and for viewing it, as is an explicit goal of HoloViews. Instead, we should be able to generalize the notion of compositors, perhaps covering the shading case above as well, allowing such overlays to be combined and shaded automatically on rich display.

These changes would make using datashading much more convenient, intuitive, and seamless in HoloViews. Tasks:

• Automatic plot sizing for aggregate()
• Shading as a plot option
• Compositors for shading categorical aggregates

[jbednar]

It would be nice to put [shade] functionality into the plot options instead so that it can be invoked automatically on rich display,

Current support in bokeh makes this approach unnecessary, so I’ve marked it “done”. There are still some gaps between Bokeh’s colormapping what shade() can do, namely using alpha to indicate value for a single color and mixing categorical colors, but those are separate issues and should be addressed outside of this one.

[jlstevens]

Just a note that new bokeh colormappers that will be better able to support datashader are being worked on: https://github.com/bokeh/bokeh/pull/8743

@kidpixo The compositor is a feature that is used occasionally internally but has never been properly documented. You might be able to get it working but I am not really sure about its status atm.