GLTF: Should joint indices be decompressed to floats?

GLTF loader has the following code with a comment:

// Load joint indices as a float array since the shaders expect float data but glTF uses unsigned byte/short.
// This prevents certain platforms (e.g. D3D) from having to convert the data to float on the fly.
else if (kind === VertexBuffer.MatricesIndicesKind) {
    accessor._babylonVertexBuffer = this._loadFloatAccessorAsync(`/accessors/${accessor.index}`, accessor).then((data) => {

This converts matrix indices unconditionally, even when they are using unsigned byte (which is the most common skinning format) - this doesn’t match the performance advice mentioned here:

https://github.com/microsoft/angle/wiki/Getting-Good-Performance-From-ANGLE#--avoid-triggering-buffer-format-conversion

Additionally, the guidelines linked seem to suggest that one-time conversion (so presumably when the vertex data is stored in a static VBO) is not a cause for concern, and it’s only relevant for 9.3 backend; right now the conversion is happening in JS which isn’t optimal memory or performance wise.

Is there any more data as to why this change was done? I’ve tracked this down to the commit that introduced it but the commit message doesn’t have any performance measurement and doesn’t indicate a specific problem this is solving.

[zeux]

… ah, bgfx doesn’t support UInt16 and so Babylon Native doesn’t either. Oh well. https://github.com/bkaradzic/bgfx/issues/739 tracks this but probably won’t get the support for a while.

And yeah UInt8 seems to run into the shader type mismatch - the shader needs to declare integer type to be able to use integer attributes as an input.

I agree that there doesn’t seem to be a straightforward solution here - the implementation is pretty simplistic. I think the right choice here is to work around this at the Babylon Native level - clearly non-normalized (or unsigned) attributes are not correctly supported by RecordVertexBuffer; I guess the only possible fix is to keep the data around and decode it on the C++ side.

I’ll close this for now since the issue is non-trivial.

[bghgary]

Right now Babylon Native implementation special cases an odd subset of Babylon JS API - it’s roughly “support types of data that valid GLTF files can produce through GLTF loader”.

It is true we are targeting loading glTF initially. The implementation will likely change a bit when we expand to more Babylon.js capabilities. I would not say Babylon Native implementation special cases anything specific to glTF though. We try not to do any processing as much as possible and in the case for joint indices, we’ve chosen to fix it in the glTF loader for now. The final implementation might be to fix it on the native side like you mentioned. I want to avoid it if there are better solutions.

The shader type confusion gets somewhat nuanced, not sure what exactly the contract here Babylon JS establishes between JS code and shader code.

The contract between JS and shader is Babylon, not WebGL. The shaders have to work for WebGL1, WebGL2, WebGPU, and Babylon Native. This contract is not super clear yet.