Does TTX merge have an upper limit of complexity? Failing to merge large COLR table into a variable font.

I am currently working on a color font project in which I generate a pretty large COLR table in TTX/XML format, then merge it into a variable font, along with a CPAL table. To be a little more specific, the base font has nearly 3000 glyphs, but then scripts are generating glyph layers, and alt glyphs, for a total of about 26K glyphs in the final font. The COLR TTX file, then, has 106K lines.

When I use ttx -m to attempt to merge the COLR ttx into a TTF, if fails with the traceback shown below.

The thing is, I was previously able to do this merge operation successfully – when using slightly fewer alternate glyphs in the font and a COLR TTX file of 90K lines. I’ve just done this again to confirm. It seems that the added alternates & added COLR/CPAL complexity are the only things really changing, here.

If this is a matter of exceeding a complexity limit, is there some way to work around that limitation?

$  ttx -m variable-font.ttf COLR-patch.ttx                                                                         
Compiling "COLR-patch.ttx" to "variable-font.ttf"...
Parsing 'COLR' table...
ERROR: Unhandled exception has occurred
Traceback (most recent call last):
  File "/Users/stephennixon/color-font-project/venv/lib/python3.9/site-packages/fontTools/ttx.py", line 401, in main
    process(jobs, options)
  File "/Users/stephennixon/color-font-project/venv/lib/python3.9/site-packages/fontTools/ttx.py", line 374, in process
    action(input, output, options)
  File "/Users/stephennixon/color-font-project/venv/lib/python3.9/site-packages/fontTools/misc/loggingTools.py", line 372, in wrapper
    return func(*args, **kwds)
  File "/Users/stephennixon/color-font-project/venv/lib/python3.9/site-packages/fontTools/ttx.py", line 292, in ttCompile
    ttf.save(output)
  File "/Users/stephennixon/color-font-project/venv/lib/python3.9/site-packages/fontTools/ttLib/ttFont.py", line 172, in save
    writer_reordersTables = self._save(tmp)
  File "/Users/stephennixon/color-font-project/venv/lib/python3.9/site-packages/fontTools/ttLib/ttFont.py", line 212, in _save
    self._writeTable(tag, writer, done, tableCache)
  File "/Users/stephennixon/color-font-project/venv/lib/python3.9/site-packages/fontTools/ttLib/ttFont.py", line 634, in _writeTable
    tabledata = self.getTableData(tag)
  File "/Users/stephennixon/color-font-project/venv/lib/python3.9/site-packages/fontTools/ttLib/ttFont.py", line 652, in getTableData
    return self.tables[tag].compile(self)
  File "/Users/stephennixon/color-font-project/venv/lib/python3.9/site-packages/fontTools/ttLib/tables/C_O_L_R_.py", line 82, in compile
    table.compile(writer, ttFont)
  File "/Users/stephennixon/color-font-project/venv/lib/python3.9/site-packages/fontTools/ttLib/tables/otBase.py", line 771, in compile
    conv.write(writer, font, table, value)
  File "/Users/stephennixon/color-font-project/venv/lib/python3.9/site-packages/fontTools/ttLib/tables/otConverters.py", line 613, in write
    value.compile(subWriter, font)
  File "/Users/stephennixon/color-font-project/venv/lib/python3.9/site-packages/fontTools/ttLib/tables/otBase.py", line 735, in compile
    conv.writeArray(writer, font, table, value)
  File "/Users/stephennixon/color-font-project/venv/lib/python3.9/site-packages/fontTools/ttLib/tables/otConverters.py", line 197, in writeArray
    self.write(writer, font, tableDict, value, i)
  File "/Users/stephennixon/color-font-project/venv/lib/python3.9/site-packages/fontTools/ttLib/tables/otConverters.py", line 492, in write
    value.compile(writer, font)
  File "/Users/stephennixon/color-font-project/venv/lib/python3.9/site-packages/fontTools/ttLib/tables/otBase.py", line 771, in compile
    conv.write(writer, font, table, value)
  File "/Users/stephennixon/color-font-project/venv/lib/python3.9/site-packages/fontTools/ttLib/tables/otConverters.py", line 278, in write
    writer.writeUShort(value)
  File "/Users/stephennixon/color-font-project/venv/lib/python3.9/site-packages/fontTools/ttLib/tables/otBase.py", line 465, in writeUShort
    assert 0 <= value < 0x10000, value
AssertionError: (65536, 'int', 16384, 'BaseGlyphRecord[]', 'BaseGlyphRecordArray')

[anthrotype]

FWIW, in the proposed COLR version 1 table, the LayerList can contain up to 4,294,967,296 unique layers since the numLayers field is a 32-bit unsigned integer.

In theory color glyphs can point to overlapping sets of layers…

Indeed, that’s true for both COLR v0 and v1 (via PaintColrLayers paint format, which defines slices of layers to be taken from the global LayerList). In FontTools, when compiling COLRv1 we try to reuse overlapping layers (see fontTools.colorLib.builder.LayerListBuilder); we should see if we can use the same (or adapted) code for COLRv0 as well

[justvanrossum]

(I haven’t run into this particular 16-bit limitation before, and tbh I was not aware of it until today.)

there isn’t some clever way to expand or exceed that limit?

I don’t think so. In theory color glyphs can point to overlapping sets of layers, but that doens’t appear a very practical thing to me. Eg. if you have layers A, B, C, D, E, F, glyph 1 could use layers A, B, C, D, E and glyph 2 could use B, C, D, E, F. I can’t think of a design in which that can be useful, though.