MergeBuffers causing "Destination array was not long enough." in _StaticBufferBuilder

Original issue: https://github.com/SteamDatabase/ValveResourceFormat/issues/346

We call the code here: https://github.com/SteamDatabase/ValveResourceFormat/blob/b277571cd47b5f41f94982170cb07e45c6c5e04a/ValveResourceFormat/IO/GltfModelExporter.cs#L438-L446

It’s throwing here: https://github.com/vpenades/SharpGLTF/blob/3c611c63f6f1cb2c9cd988faa4e01e53b11cc095/src/SharpGLTF.Core/Schema2/gltf.BufferView.cs#L386

I tested with WriteSettings.MergeBuffers = false and it exports, it also seems to not happen with other models.

I don’t know if this is a mistake on our end or what, but we don’t call that Save multiple times, so this shouldn’t be a threading issue. That particular model has 732 buffers, so it could be something to do with that.

Stacktrace:

System.ArgumentException: Destination array was not long enough. Check the destination index, length, and the array's lower bounds. (Parameter 'destinationArray')
   at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length, Boolean reliable)
   at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length)
   at System.SZArrayHelper.CopyTo[T](T[] array, Int32 index)
   at System.Collections.Generic.List`1.InsertRange(Int32 index, IEnumerable`1 collection)
   at SharpGLTF.Schema2._StaticBufferBuilder.Append(Byte[] data)
   at SharpGLTF.Schema2.BufferView._IsolateBufferMemory(_StaticBufferBuilder targetBuffer)
   at SharpGLTF.Schema2.ModelRoot.MergeBuffers()
   at SharpGLTF.Schema2.WriteContext._PreprocessSchema2(ModelRoot model, Boolean imagesAsBufferViews, Boolean mergeBuffers)
   at SharpGLTF.Schema2.WriteContext.WriteTextSchema2(String baseName, ModelRoot model)
   at SharpGLTF.Schema2.ModelRoot.SaveGLTF(String filePath, WriteSettings settings)
   at SharpGLTF.Schema2.ModelRoot.Save(String filePath, WriteSettings settings)
   at ValveResourceFormat.IO.GltfModelExporter.WriteModelFile(ModelRoot exportedModel, String filePath) in ValveResourceFormat\IO\GltfModelExporter.cs:line 445
   at ValveResourceFormat.IO.GltfModelExporter.ExportToFile(String resourceName, String fileName, Model model) in ValveResourceFormat\IO\GltfModelExporter.cs:line 293

Version: SharpGLTF.Toolkit 1.0.0-alpha0022

[vpenades]

@xPaw In https://github.com/vpenades/SharpGLTF/commit/4788e4715edffa5fc6a3027950964dd06fa14d13 I’ve just added a few improvements to handle scenarios with very large models.

When merging to a single buffer, it checks whether the total sum of buffers is less than 2Gb, otherwise it throws a NotSupportedException.

Also I’ve added new write settings to support merging the buffers up to a given size, splitting the resources into multiple bigger but valid sizes. This allows writing very large models, while keeping a short number of binary files.

An example of that configuration can be found here.

So, right now the approach to write a GLB would be:

• If the resources sum less than 2Gb, write GLB
• Else, write glTF, using the new buffer merging/splitting settings.

Let me know if this allows you to handle these large models.

[vpenades]

The difference between glTF and GLB is that glTF can have multiple buffers, and GLB only one. But each individual buffer is limited to what can be addressed by a 32bit value.

Furthermore, .NET uses a SIGNED Int32 value for most addressable collections. I’ts widely used in Arrays, Lists, ArraySegments, Spans, etc.

This means that in practice, the number of bits that can be used for addressing is just 31 bits, which means only 2Gb can be addressed.

And again, I suspect many implementations might have the same issue, so it could be said the largest a buffer can grow without risking having trouble is just 2Gb.

Allowing SharpGLTF to grow up to 4GB might require changing lots of Ints by UInts, and writing custom collections that supports large buffers. Just replacing List<T> by a LargeList<T> looks like a lot of work.

My rule of thumb would be: whenever a glTF is bigger than 1gb, store it as a zipped glTF, instead of a GLB