WebGL renderer?

EDIT: Tracking progress:

• Bitmap + vector sprites
• Speech bubbles
• Pen layer
• Sprite is touching mouse
• Sprite is touching sprite
• Effects
• Layer ordering
• Rotation style
• Convex hull generation / “if on edge, bounce”
• Sprite is touching color
• Color is touching color
• Resizable stage
• Subpixel positioning of SVG viewboxes

Incorporating WebGL in some form is necessary to implement graphic effects at a reasonable speed.

I was thinking of rewriting the renderer to use WebGL at its core, similar to scratch-render. This could potentially take care of a number of currently missing blocks in one fell swoop:

• Effect blocks
• Layering
• Touching color
• If on edge, bounce (requires a “precise” bounding box, and hence convex hull generation, for Scratch-compatible results)

There are a couple of implementation details I’m not sure about, though:

In regular Scratch, each sprite has a fixed number of costumes, and you can only create costumes by clicking a button in the editor. In scratch-js, on the other hand, you can dynamically instantiate as many Costume objects as you like.

This presents a problem because WebGL textures are never garbage-collected. In Scratch, the textures that back a costume are deleted when the costume is. In scratch-js, though, you could (as far as I can tell) do something silly like create and render a new Costume object every frame.

I can think of three ways to deal with this:

1. Add a destroy method to Costumes that you require API consumers to call.
2. Maintain a list of every created texture in the Renderer. Every rendering step, mark every texture that ends up getting rendered, then sweep away those that aren’t.
3. Give each texture a refcount. At every call site that adds/removes textures (e.g. the Sprite.costume setter), update the refcount and delete the texture if the count is 0.

I’m leaning heavily towards the second option myself–it seems like the least complex one.

The other implementation detail that might be affected is pen color, which is also being discussed in #22. While the canvas API accepts any CSS color string, WebGL requires colors in an RGBA format. That may factor into the pen color discussion somewhat, as every accepted pen color format will need to be converted into RGB(A).

I’d love to hear your thoughts on all of this–this is a really promising project!

[adroitwhiz]

I’d hope not–I’ve already started in on it!

If you look inside Renderer.mjs and the renderer folder, you can get a feel for the complexity WebGL adds. You’re correct that it’s somewhat more difficult to understand than the canvas API, especially if you’ve never seen it before, just because it’s “closer to the metal”, but I think that it’s a fair tradeoff given how much more you can do with it.

I’ve tried to abstract away the more boilerplate-y stuff (in classes like ShaderManager) and I’ll probably clean it up more as I progress. I also haven’t yet heavily commented the code.

If this is your first time looking at WebGL code, I’d recommend looking at WebGL Fundamentals or the MDN tutorial to get a feel for how the API functions.

The reasons I don’t want to go with Pixi are:

1. It’s a pretty big library (360 KB minified).
2. It contains a lot of APIs that scratch-js will never use.
3. You’d have to manually maintain the correspondence between Pixi APIs and scratch-js APIs. This includes chores like copying all transform attributes from scratch-js Sprites to PIXI.Sprites, and managing the creation and deletion of PIXI.Textures (which, to be fair, you also need to do in WebGL).

[adroitwhiz]

I considered Maps too, but they can’t be reordered which kinda defeats the purpose of using them for layering.