Additional debugging information when a DrawCommand fails

Some background: I have a big, complex application built on Cesium. One of the systems we load data from recently had a major upgrade and now our application is crashing, and I can’t for the life of me figure out why. I’m trying to reproduce a test case, but I think if I knew how to reproduce it in a simpler context, I could probably figure out how to stop it from happening in the first place.

Each time it crashes, I get a stack trace that points to this line in Context.js, which I managed to ascertain is because drawCommand._vertexArray has been destroyed. (drawCommand.owner is a BillboardCollection, if that helps.) It sounded a lot like #6916 but a) I’m already using 1.49 which includes the fix for that issue, and b) I’ve actually seen several other issues with very similar stack traces even though the root causes were wildly different.

That brings me to what I actually hope to achieve here. Yes, I would love some suggestions of what my next steps should be trying to debug this, but more than that I’d like some capability to generate a more helpful message than “somebody destroyed this at some point, for some reason”. Is this already possible? Could it be added, probably behind a debug flag, for performance reasons?

[thw0rted]

Another update: I finally got it bulletproof, but at a cost. If I totally remove all billboards, labels, and points (basically anything 2D) from the entities I make, it never crashes. Just to really be sure, I turned the Billboard into a BoxGraphics with an ImageMaterial using the same texture – works great, except of course that I can’t size it dynamically relative to the viewport.

The absolutely infuriating part is that on my development system, everything is fine, but working against the upgraded (production) system, on an isolated network, I get these crashes that I can’t otherwise reproduce. I’ll keep poking at it next week. I’m planning to try bringing over a build based on 1.48 (maybe even 1.47?) just to be certain it’s not something introduced when I upgraded to 1.49.

[thw0rted]

Small update: just checking for _vaf.isDestroyed() in BillboardCollection didn’t get rid of all the crashes, but adding if (va.isDestroyed()) { return; } at the beginning of continueDraw appears to have stopped all crashes. Again, I need some time to test and make sure it’s totally stable, but same questions from the previous post apply: is this a bad addition? Will it hurt performance? It feels like a band-aid over a wound of unknown origin, so I’d really like to figure out what this race condition is.

A thought occurred to me that might be important: my dev system is comparatively powerful, and we probably have somewhat less test data than what’s on the production system (fewer Entities loaded). I’ll try to add a bunch of extra test data and see if performance factors into this – if picking takes many frames to compute, maybe that accounts for the “stale” command with the deleted VAF?