Discussion: HarfBuzz shaping extension design

In #5244 I have mentioned wanting to create HarfBuzz-based font shaping extension for libGDX. Since then I’ve been working on it, in my limited spare time. I have already created a JNI-Gen binding for HarfBuzz, and now I’m designing the shaper itself. It is not yet in a PR form, but I want to discuss some things I have encountered so far and how to tackle them.

First off, current BitmapFont is not well suited for being part of this, so I am creating a new “Font stack” around FreeType and HarfBuzz, which may eventually replace current one, if successful. BitmapFont does a lot of things which are not really needed in the new system, and more importantly assumes some things which are no longer true in complex layout, for example, that glyph ID maps 1:1 to char.

Separation of glyph generation and shaping

Even if the new font stack would be better than old one in every way, it could never replace it, unless it supports all platforms. And in this respect, the only one that is problematic, is GWT. It could be possible to emscripten all native libraries for JS, but it is not something I would really want to do. And since we’re probably not dropping support for GWT either, I am planning to split the new stack around these three responsibilities:

• Shaping (with implementations: )
  • HarfBuzz
  • current approach, enough for Latin scripts, possibly extended to support widely used scripts like Arabic
• Supplying glyphs to shaper
  • FreeType creation on demand (“incremental”)
  • Pre-rendered, cached from incremental, or even hybrid
• Rendering shaped runs to (Sprite)Batch
  • BitmapFontCache reimplementation
  • maybe DistanceFieldFontCache also?

This division feels more appropriate than BitmapFont/BitmapFontData, which now do “too little and too much at the same time” respectively, as BitmapFont is basically just a wrapper for BitmapFontCache + BitmapFontData, and BitmapFontData is too tightly coupled to shaping and to the idea of pre-rendered glyph pages.

While on this topic: I could be wrong in my judgement, but I’d like to simplify the new equivalent of BitmapFontCache. Does it really need the ability to add multiple unrelated text layouts and to render only a part of the text? Latter will become almost impossible to implement fast & correct with the complex layout which can omit, add or even reorder characters.

Scaling and flipping

I don’t think that font should care about its scaling (BitmapFontData.setScale()) nor direction of Y coordinate (BitmapFont.isFlipped()). These things should be handled in layer one above (possibly in aforementioned BitmapFontCache) - currently it only complicates already pretty complicated code. IMO font should have only 2 size-related settings:

• size in pixels - this is what shaping works in (HarfBuzz works only on integers anyway) and this is also the size of glyphs on backing texture. Both things should be pixel perfect for pretty results.
• size in points - this is what (for example) UI system that uses the font works in. Ideally, these sizes would match on standard screen and be multiples of each other on high DPI screens (e.g. 100 points = 200px on 2x retina).

Maybe it would be best to specify only (float) size in points and (float) scaling factor to get size in pixels (int) from it. But I’m not sure on this yet.

Unicode

HarfBuzz does shaping, but does not help with BiDirectional text layout, nor deciding where line should be wrapped, nor between which chars can editing caret go (which seems to be a surprisingly tricky problem in HarfBuzz-ed unicode). Thankfully, all this can be solved with ICU library. However, it weighs around 12MB, which is quite a lot, but it should be possible to trim it as it contains a lot of unnecessary data (for this purpose).

Input text

While the current markup [BLUE]language[] is easy to use, it does not play well with most parts of the text stack. So for this font stack, I am playing with the other approach (used at least in iOS and possibly in Android): annotated text, i.e. object that associates text characters with their attributes.

This should make it easier to generate colors programatically, as text does not have to be modified anymore, and it will allow for colored text in input text fields, as markup tags previously got in the way and messed up mapping between GlyphRun indices and source text indices. Converter from current markup strings to annotated strings is, of course, planned.

Above, I mentioned attributes and not just color - I am playing with mixing different fonts in as well. This not only means multiple different typefaces in single string, but also italics, bolds, underlines (courtesy of FreeType) and also different sizes.

I am also considering to add a special support for tabulators (\t). Initially they will probably work like in terminal console (“move the run beginning with ‘\t’ to the right up to the nearest multiple of 4 spaces worth of space”) but it would be pretty easy to add support for more complex behavior, customizable when invoking the text layout.

---

What are your thoughts? Are there any other things that should be changed about the current system? Some things that I want to omit which are actually very useful? Or some other features which were overlooked, but should be factored in from the start?

I’ll keep this issue open until I have a PR ready.

[NathanSweet]

Sounds cool! Our text area and even text field support has always been a bit janky. I wish you the best of luck, I know it’s a huge project.

[Darkyenus]

My repo is here and while it works as a proof of concept, it is not yet finished and ready for general use. The main problems from the usability point of view right now are:

• No integration into existing font infrastructure (no scene2d, etc. integration)
• No pre-built artifacts of natives (HarfBuzz)

And the code is not 100% finished either, although most of the “hard problems” are already solved. So it is not something any random user can just start using. But it can serve as a robust base for any further development, if you are prepared to put some effort in.

I am currently not planning on working on it further, but I will be able to help and answer questions about it, should somebody want to continue working on it.