Use ImageBitmaps for GLTFLoader (and possibly others)

TL;DR: I’d like to propose that ImageBitmapLoader be used by default (with a fallback to ImageLoader for unsupported browsers) for at least the glTF loader and possibly others that have appropriate restrictions on their input images.

I’ve been doing some experiments to try and reduce framerate hitches in https://xrdinosaurs.com, especially in VR. The biggest issues I’ve seen are that when you load in a new model (all binary glTF files with Draco compression and embedded images) there was always a long and noticable hitch in the framerate immediately before the newly loaded dinosaur displayed for the first time.

Turns out one of the biggest blocking operations on the main thread is the internal image decode that happens in texImage2D. Here’s a trace of a similar situation recorded from the webgl_loader_gltf.html example:

(Measurements taken on my desktop PC: Intel Core i5 3.3Ghz with a GTX 1660 GPU)

The really big column to the right is the first frame where the loaded glTF is going to be displayed. Some of the largest time sinks in that rather large operation are the dark green blocks at the bottom, which are the aforementioned image decodes. They take 103.29ms, 24.81ms, 30.23ms, 16.07ms respectively, each of which is blocking the main thread. The entire task (frame) takes 448.22ms (28 frames @ 60Hz, 40 frames @ 90Hz VR). This is enough to cause a noticeable stall in any VR experience.

ImageBitmaps, if you’re not already familiar with them, were designed to allow image decoding to happen pre-emptively off the main thread to produce images that are “ready to use” by the GPU. gl.texImage2D() accepts them, and will frequently take significantly less time to complete because the decode step doesn’t need to happen synchronously.

I made a simple switch inside TextureLoader.load() to allow ImageBitmapLoader to be used instead of ImageLoader if supported:

// Current code
//var loader = new ImageLoader( this.manager );

// New code
var loader = null;
if (typeof createImageBitmap !== 'unknown') {
  loader = new ImageBitmapLoader( this.manager );
} else {
  // Because some browser still don't support ImageBitmaps.
  loader = new ImageLoader( this.manager );
}	

And then ran the same example again:

You can see now that the rightmost task is much shorter now (260.13ms in total, nearly cut in half!) The image decoding blocks are still there, but they’re happening in a thread pool off the main thread. The initial frame the mesh is first displayed in is delayed because it’s waiting on the decode in the other thread, but because that frame now completes faster the user perception is that it takes about the same amount of time with the main difference being that the scene stays more responsive while the load is happening. The difference is more pronounced on https://xrdinosaurs.com where the stall now registers as a minor blip rather than an uncomfortable and lengthy stall.

Of course, nothing is ever that simple, is it? Turns out this only works for glTF files or similar formats where the texture usage is well defined. In other cases where the textures are being applied to dynamically generated meshes they have an annoying tendency to come in upside down because ImageBitmap texture sources ignore the gl.UNPACK_FLIP_Y_WEBGL parameter, and the flipY option that they do support must be set at load time, while THREE.Texture allows it to be set at any time. As a result a straight universal replacement of ImageLoader doesn’t work. In the context of GLTFLoader specifically, however, we know that the textures will never need to be flipped because the format dictates that’s the case. So it’s safe to use ImageBitmapLoader in that context while continuing to use ImageLoader everywhere else. I imagine this would be the case for many model formats, really, I just don’t have a personal motivation to examine anything other than glTF at the moment.

You can see the code change I applied to my project here.

[toji]

Okay, so my Safari issue was due to a really dumb mistake on my part. 😓

Good news is, though, that after fixing that my code appears to be working on every browser I’ve tested.

[mrdoob]

Turns out this only works for glTF files or similar formats where the texture usage is well defined. In other cases where the textures are being applied to dynamically generated meshes they have an annoying tendency to come in upside down because ImageBitmap texture sources ignore the gl.UNPACK_FLIP_Y_WEBGL parameter, and the flipY option that they do support must be set at load time, while THREE.Texture allows it to be set at any time.

Maybe it’s time to move away from gl.UNPACK_FLIP_Y_WEBGL and do the flipping in glsl: 1.0 - uv.y.