ArrayBuffers as data source

We (@bbss and @dcj) are using deck.gl with great success for a web-app to visualize aircraft flights, in support of a community advocacy project to quantify, analyze, report, and visualize the affects of aircraft noise.

The app downloads data about the flights and their reported positions for 24-hour periods, on a typical day there are ~4K flights and ~100K positions (time, lon, lat, altitude), and when the user specifies a range of dates, each day is downloaded in sequence (as needed).

The app supports a number of viewing options: all flights for one day animate all flights over selected time-range with “time-lapse” control sequence (per hour or per day) over a selected period This makes for a lot of data in the browser, and we’ve had to work hard to optimize for performance, we now offload the data fetching and filtering to web-workers.

In order to transfer our data between each of the web-workers and then on to the main browser JS thread, we use the Transferable Objects APi, and Array Buffers.

We’ve gotten pretty far with this approach, but we haven’t been able to figure out how to pass our ArrayBuffers to deck.gl as layers, so for now we have to convert our ArrayBuffers back to a regular JavaScript list.

The documentation seems to suggest that passing an ArrayBuffer to deck.gl layers is possible, we have some questions and suggestions about that:

What is the status/schedule/roadmap for the binary-data RFC?

https://github.com/uber/deck.gl/blob/master/dev-docs/RFCs/v6.x/binary-data-rfc.md

And with respect to Manual Buffer Management:

http://deck.gl/#/documentation/developer-guide/writing-custom-layers/attribute-management?section=advanced-topics

We attempted to use this, and were able to pass a buffer so that isExternalBuffer was set and got that to display something, however going so low level to calculate each of the attributes manually was difficult. We had hoped that there would be finer-grained control as to with what kind of chunks the accessors get called. We thought by implementing getNumInstances and by having the correct size on the attributes, the iterator could create the right size chunks but that doesn’t seem to be the case…. Is there a way to iterate over ArrayBuffers as you would over a regular collection and still have the layers use the accessors?

We realize that laying out binary data is very specific to each application, and that it is difficult to create a general approach in a framework. In our case we have something like [numOfEntities(2), entity1, entity2, numOfEntities(3) entity-1, entity-2, entity-3, numOfEntities ..etc..]. Additionally ArrayBuffers might have different byteLengths within them to further complicate things, the ideal solution would support this.

It would be useful to have at least some way to iterate a fixed step and byteLength on an ArrayBuffer and still be able to use the specific layers functionality. Partitioning/converting an ArrayBuffer from a WebWorker into regular JS has a significant cost. Do you have any plans for providing a way to specify how to iterate in order to still use the layers with ArrayBuffers?

[bbss]

@Pessimistress

Thank you so much for the expansive answer. The createIterable utility was very useful, we’ve copied and adjusted it a bit to support partitioning for the path layer.

By pre-calculating multiple layers and switching visibility (also per your recommendation), the iterable solution is fast enough for our use-cases so directly passing buffers was not yet necessary. It’s good to now know how it can be done though, in case we ever do need even more speed.

No more questions on this issue on our side. Thanks again!

[Pessimistress]

The lack of documentation is somewhat intentional - the solutions I’m going to list below are all somewhat hacky here and there. We did not want to promote them until we have confidence in the API design.

With that said, here are some techniques that may help your use case:

Create attribute buffers manually

In the worker you’ll pack your data into typed arrays that the layer is expecting:

// point[0].x, point[0].y, point[0].z, point[1].x, point[1].y, point[1].z, ...
const positions = new Float32Array(...);
// point[0].r, point[0].g, point[0].b, point[0].a, point[1].r, point[1].g, point[1].b, point[1].a, ...
const colors = new Uint8ClampedArray(...);

// send back to main thread
postMessage({positions, colors}, [positions.buffer, colors.buffer]);

Once the main thread receives the data, you can use it like:

// `data` is received from the worker
new PointCloudLayer({
  // since there's no raw data, the layer needs to know how many instances to draw
  numInstances: data.positions.length / 3,
  // external buffer
  instancePositions: data.positions,
  instanceColors: data.colors,
  // constant accessor works without raw data
  getNormal: [0, 0, 1]
});

As you pointed out, it is a lot to ask of the users of this library to understand the buffer layout of each layer. However, if absolute efficiency is what you’re after, by skipping the internal creation and population of new buffer arrays, this is the most performant way to get data onto the screen.

Create an iterable object from binary

Instead of a plain array object, you can also supply anything that implements the iterable protocol to the data prop. This is probably the most promising solution for your custom binary layout.

I created a proof of concept to show how this works.

If this turns out to be useful, maybe we could incorporate the createIterable utility into the deck.gl core, and allow you to provide a “iterable-like” object to data.