Unicode: surrogate pairs encoded invalidly after x number of chars

The problem here is that the character Kappa starts being encoded as xml entities, unfortunately this is an non valid character encoding. I don’t understand why this is happening and why it happens after X characters instead of at any point.

@Test
public void tooManyKappas()
	 throws XMLStreamException
{
	XMLOutputFactory factory = OutputFactoryImpl.newInstance();
	if (factory instanceof OutputFactoryImpl) {
	     ((OutputFactoryImpl) factory).configureForSpeed();
        }
        //loop to find exactly at which point entity encoding kicks in.
	for (int j = 0; j < 1000; j++) {
		final ByteArrayOutputStream baos = new ByteArrayOutputStream();
		XMLStreamWriter writer = factory.createXMLStreamWriter(baos, StandardCharsets.UTF_8.name());

		final String namespace = "http://example.org";

		StringBuilder kappas = new StringBuilder();

		for (int i = 0; i < (2000 + j); i++) {
			kappas.append("𝜅");
		}
		writer.writeStartElement("", "ex", namespace);
		writer.writeCharacters(kappas.toString());
		writer.writeEndElement();
		writer.close();

		assertEquals("fails at " + (2000 + j),
			    "<ex>" + kappas + "</ex>",
			    new String(baos.toByteArray(), StandardCharsets.UTF_8));
	}
}

I hope this minimized test case is off help. It’s definitely due to something internal to aalto. WSTX does not have this issue (or at a much higher loop number…).

The problem really is that aalto-xml reader does not deal with its own output in this case. Which is correct as its the writer that is wrong.

[cowtowncoder]

@JervenBolleman happy to fix it – it has been a while as I am not that active with xml tools these days, but feels good to clean up the backlog. Aalto is not nearly as well tested as Woodstox is, but would be great to get it to same level of quality.

[JervenBolleman]

Thank you so much for picking this up. It will be great to use aalto-xml again, it’s a real speed boost! I really appreciate it that you took the time to look into this.