Discuss: Add support for 2D density visualizations

Hi all,

I would really like to expand Vega-Lite with support for 2D density representations, leveraging Vega’s kde2D, isocontour, and heatmap transforms. Adding these would make Vega-Lite largely “feature complete” for my current visualization teaching needs, and make it more comparable to other popular plotting libraries. However, it is not clear how to best do this.

I see two general approaches:

1. Use existing mark types only (geoshape for contours, image for heatmaps) and leverage new transforms (density2d, contour, and heatmap) to generate the appropriate input data to the mark. This is the same approach we’ve followed so far when adding transforms such as density and regression. However, Vega’s kde2d and heatmap transforms both interleave data-space and encoding-space concerns in ways that make this difficult. For example, the kde2d transform takes x and y field accessors that must return pixel-space values. Typically this is done using an expression that maps an underlying data field through a defined scale transform. So, for this to work in Vega-Lite we need a way to generate / access appropriate scale transforms, introducing a cross-cutting concern. Moreover, we’d like Vega-Lite to also use those scales to add appropriate axes, so we’d also have error-prone redundancy in the specification if we need to provide x/y fields in both the transform and encoding. (This is further complicated by the fact that geoshape doesn’t take x and y encodings anyway…) The heatmap transform has a separate issue, which is that it accepts expressions for determining pixel color and opacity. While these can be stand-alone, most times we actually want to use a defined color scale (and corresponding legend), again mixing transform and encoding concerns.

2. An alternative is to create new mark types, such as contour and heatmap. Ideally these marks could accept either pre-calculated raster grid data (from which contours / heatmaps can be directly generated) or point data (to which the kde2d transform would be applied). The Vega-Lite compiler would need to generate appropriate transforms and encodings. I imagine transform parameters could be passed as mark properties. There are still some limitations to this approach: in Vega we can separately generate heatmap images and then use them as input to image marks, such that we could in theory do things like create an entire scatter plot where each point is a small density heatmap. However, I don’t think this extra expressiveness is critical for Vega-Lite. The biggest hurdle to this approach is that I have no idea how to implement it in the current VL compiler. So I can’t estimate the feasibility or difficulty. That said, I would be happy to collaborate with someone more knowledgable.

While neither solution is completely satisfactory, I’m leaning towards the approach of adding new mark types. I think the interleaving of data-space and encoding-space operations in VL transforms breaks too much, in terms of both output and user mental models.

Any thoughts or feedback, particularly relative to the feasibility of option 2?

[metasoarous]

Just wanted to chime in that I’m super excited to see this issue getting some attention, and would find this feature extremely useful. Thanks so much for thinking about it!

FWIW, option 2 sounds ideal from a user standpoint. Sounds like it makes things simpler on the implementation side as well. IIUC, option 1 might in theory offer more flexibility/leverage in certain cases? Seems not critical, however, and may be fine to say “if you need more than this, see Vega.”

Should this issue be considered as superseding #1919?

Again, thanks for all of you work on this!

[kanitw]

Hi Jeff,

Thank for the proposal.

I think I agree we should should pursue option 2 since post-encoding transform isn’t a primitive that we provide in VL. It’s also likely produces a much more concise specification.

I think it should be doable (I don’t see why it wouldn’t), but would might a a few days to investigate / prototype. Do you have a specific time frame you’d like to have this by? (I’m thinking after VIS deadline would be a good time to investigate more.)

The parameters above makes sense for the most parts, but I still have some comments / questions:

1) Raster as mark property or encoding channel?

The raster property currently accepts a field name. It’s worth noting that we never accepts a field in a mark property before. So this diverges from a former pattern that mark property only accepts value directly, but encoding channel can accepts a field.

For consistency, it might be better to support raster field via an encoding channel?

2) When is kde triggered? (raster = false)

If I understand correctly, it seems like raster will be automatically true (no kde2d) or false (apply kde2d), based on whether the x/y encoding channel (input for kde2d x/y) are specified?

(I think this makes sense, but it’s not explicitly clear from the proposal above.)

3) Heatmap Color Expr?

From the proposal above, it’s still a bit unclear how the color encoding for heatmap:

vl.color(),     // -> kde2d groupby, heatmap color

could generate an appropriate expression like

"color": {"expr": "scale('density', datum.$value / datum.$max)"},

in https://vega.github.io/vega/examples/density-heatmaps/

To brainstorm, I could see we add expr support to the color channel for just heatmap like this:

encoding: {
  ...,
  color: {
    expr: "datum.$value / datum.$max"
    scale: ... // optional scale customization
    legend: ... // optional legend customization
  }
}

Note that we currently have experimental signal support for color as well, but I think this is a bit different since datum in the expr isn’t really the origin datum (data point) on the data source that we directly seed in the the Vega-Lite marks anymore.

4. Scattered Heat Map Case

There are still some limitations to this approach: in Vega we can separately generate heatmap images and then use them as input to image marks, such that we could in theory do things like create an entire scatter plot where each point is a small density heatmap. However, I don’t think this extra expressiveness is critical for Vega-Lite.

If we want to support this, I could see we support special imageX/Y channels (or something similar) for heatmap marks. We can then map imageX / imageY to the final image mark’s x/y encoding channels.

We probably shouldn’t do this in the initial implementation, but I’m just mentioning here that it’s not totally impossible.