Proposal for WebGL rendering

I’m working with @jmuhlich and @dpwrussell on an application using OpenSeadragon. Along the way, we’ve outlined one approach to a WebGL rendering engine within OpenSeadragon. We’d like to know whether these changes would be of general interest, and we’re looking for feedback on the general approach outlined below.

---

We propose changes to OpenSeadragon to replace canvas 2D rendering with rendering through WebGL. The proposed OpenSeadragon updates will allow application-specific “GLSL” fragment shaders to modify all pixels in tiled images. Custom shaders will let applications render color images directly from a variety of useful scientific encodings such as 2D label maps or high dynamic range grayscale images.

All browsers supporting OpenSeadragon’s use of canvas also support the proposed use of WebGL. No up-to-date browsers will benefit from a 2D canvas fallback from WebGL; therefore, we propose removing “2d” contexts entirely to reduce the number of implementations to maintain. After feature detection, OpenSeadragon will use either a “webgl2” context, a “webgl” context, or the HTML rendering fallback.

If OpenSeadragon detects WebGL support, each image record will manage TypedArray objects from decoded ajax responses or from rendered image elements. Methods of the drawer, tiled images, and tiles will render these arrays to textures and bind all textures per frame to a webgl context. Without WebGL, the image record will still manage image elements directly. In either case, no rendered 2D canvas context will be created or stored by the image job. With WebGL, the drawer will call a new method of Tile named drawGL, which will use a cached TypedArray. Otherwise, the drawer will still call drawHTML from a cached image element. In either case, the drawCanvas method will not be called, and the sketch canvas will not be needed.

We will add to the API, updating the interfaces and descriptions in the OpenSeadragon docs. The API will remain compatible with the current (version 2.3.x) API; however, plugins will break if they rely on overwriting methods that Openseadragon will no longer call internally. New external API endpoints will allow projects depending on OpenSeadragon to supply custom functions written in GLSL ES 1.00 to use in all current browsers and/or GLSL ES 3.00 to use in browsers with detected webgl2 support.

Open Questions:

• With Ajax enabled and WebGL supported, we intend to store pixel data in TypedArrays instead of HTML image elements to support high dynamic range grayscale images common in scientific imaging. As HTML image elements only provide 8-bit pixel data from standard HTML5 image formats, we will need to support loading from higher-depth formats through JavaScript implementations of PNG (16-bit), TIFF, etc. We are unsure whether to bundle these decoders within OpenSeadragon or ask applications to pass a custom decoder as a tilesource option.

• OpenSeadragon will cease to use certain documented properties and methods. For example, the viewer event tile-loaded supplies an HTML image element. Even when no image element is needed internally, the new implementation of such events must always supply an image element and handle modifications to it in order to maintain compatibility with existing plugins. The documentation only references some properties (e.g. a drawer’s SketchCanvas) indirectly within descriptions of other properties and methods. We are unsure whether to remove such properties entirely.

• We suggest continuing to handle the pyramids generated by an ImageTileSource by initializing tiles with “2d” contexts, but these contexts will be used to either call drawGL on a TypedArray or drawHTML on a HTML image element. We are unsure whether any simpler solution could better support the API.

[yellowtailfan]

I am considering coding something similar in purpose to what @thejohnhoffer has proposed. I have similar needs:

• View multi-channel high bit depth (uint16, float16, or float32) image data
• Mapping from multi-channel high bit depth to 8-bit RGB for display
• User interface for making interactive adjustments (e.g. gamma, offset, gain, range, normalise) to the mapping quickly on the client side without re-fetching image data from the server

Openseadragon and viaWebGL look like excellent starting points for this.

I have a concern about the performance of using pixelwise loops in JavaScript for data conversions. I found an example from a LeafletsJS plugin which does this kind of pixelwise data conversion in GLSL.

Are there any status updates on the work proposed in this issue? It would be good to know before I start coding.

[iangilman]

Nothing new on this front. All of the work on OSD is from volunteers, so we don’t have a roadmap… It’s just whatever someone is motivated to work on. I feel like there’s potential for the many institutions that use OSD to kick in some money for more directed development (and we do have an Open Collective), but I think in order to make that happen, we’ll need someone who can actually go around and hit those institutions up. I’m thinking it may be time to start looking for that person, so if you (or anyone else) have suggestions, I’m all ears!

But I digress. No, nothing new on the WebGL front. Every now and then someone gets interested and takes a stab at it, but none of it has come to fruition yet. A company almost hired me to do it a year or two ago, but that didn’t end up happening either. I think I’ve got a pretty good idea of how to make it happen, but it’s a big job and I can’t afford to take that much time off from work without getting paid for it.

If you’re interested in tackling it, or if you know someone who might want to fund it, dive in! I’m happy to help point the way 😃