Allow users to register custom Kryo serializers

Version: release/os/4.4

Problem

We are using persistent immutable maps which are in scope when one of our flows suspend, causing it to be serialized by Kryo. The deserializer fails, causing the following exception to be emitted to the log:

[ERROR] 11:39:49,636 [FiberDeserializationChecker] interceptors.FiberDeserializationChecker. - Encountered unrestorable checkpoint! [errorCode=ewoky4, moreInformationAt=https://errors.corda.net/OS/4.4/ewoky4]
 com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException
Serialization trace:
dataObject (co.paralleluniverse.fibers.Stack)
stack (net.corda.node.services.statemachine.FlowStateMachineImpl)
	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.serializers.CompatibleFieldSerializer.read(CompatibleFieldSerializer.java:145) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:782) ~[kryo-4.0.2.jar:?]
	at co.paralleluniverse.io.serialization.kryo.ReplaceableObjectKryo.readObjectOrNull(ReplaceableObjectKryo.java:107) ~[quasar-core-0.7.10-jdk8.jar:0.7.10]
	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:132) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543) ~[kryo-4.0.2.jar:?]
	at co.paralleluniverse.fibers.Fiber$FiberSerializer.read(Fiber.java:2137) ~[quasar-core-0.7.10-jdk8.jar:0.7.10]
	at co.paralleluniverse.fibers.Fiber$FiberSerializer.read(Fiber.java:2067) ~[quasar-core-0.7.10-jdk8.jar:0.7.10]
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:813) ~[kryo-4.0.2.jar:?]
	at co.paralleluniverse.io.serialization.kryo.ReplaceableObjectKryo.readClassAndObject(ReplaceableObjectKryo.java:112) ~[quasar-core-0.7.10-jdk8.jar:0.7.10]
	at net.corda.node.serialization.kryo.KryoCheckpointSerializer$deserialize$1$1.invoke(KryoCheckpointSerializer.kt:83) ~[corda-node-4.4.jar:?]
	at net.corda.node.serialization.kryo.KryoCheckpointSerializer$deserialize$1$1.invoke(KryoCheckpointSerializer.kt:33) ~[corda-node-4.4.jar:?]
	at net.corda.node.serialization.kryo.KryoStreams.kryoInput(KryoStreams.kt:20) ~[corda-node-4.4.jar:?]
	at net.corda.node.serialization.kryo.KryoCheckpointSerializer$deserialize$1.invoke(KryoCheckpointSerializer.kt:72) ~[corda-node-4.4.jar:?]
	at net.corda.node.serialization.kryo.KryoCheckpointSerializer$deserialize$1.invoke(KryoCheckpointSerializer.kt:33) ~[corda-node-4.4.jar:?]
	at net.corda.node.serialization.kryo.KryoCheckpointSerializer$kryo$1.execute(KryoCheckpointSerializer.kt:61) ~[corda-node-4.4.jar:?]
	at com.esotericsoftware.kryo.pool.KryoPoolQueueImpl.run(KryoPoolQueueImpl.java:58) ~[kryo-4.0.2.jar:?]
	at net.corda.node.serialization.kryo.KryoCheckpointSerializer.kryo(KryoCheckpointSerializer.kt:57) ~[corda-node-4.4.jar:?]
	at net.corda.node.serialization.kryo.KryoCheckpointSerializer.deserialize(KryoCheckpointSerializer.kt:71) ~[corda-node-4.4.jar:?]
	at net.corda.node.services.statemachine.interceptors.FiberDeserializationChecker$start$2.invoke(FiberDeserializationCheckingInterceptor.kt:101) ~[corda-node-4.4.jar:?]
	at net.corda.node.services.statemachine.interceptors.FiberDeserializationChecker$start$2.invoke(FiberDeserializationCheckingInterceptor.kt:51) ~[corda-node-4.4.jar:?]
	at kotlin.concurrent.ThreadsKt$thread$thread$1.run(Thread.kt:30) ~[kotlin-stdlib-1.2.71.jar:1.2.71-release-64 (1.2.71)]
Caused by: java.lang.NullPointerException
	at com.github.andrewoma.dexx.kollection.MapAdapter.put(ImmutableMap.kt:57) ~[kollection-0.7.jar:?]
	at com.github.andrewoma.dexx.kollection.MapAdapter.put(ImmutableMap.kt:45) ~[kollection-0.7.jar:?]
	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:162) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:731) ~[kryo-4.0.2.jar:?]
	at co.paralleluniverse.io.serialization.kryo.ReplaceableObjectKryo.readObject(ReplaceableObjectKryo.java:92) ~[quasar-core-0.7.10-jdk8.jar:0.7.10]
	at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:391) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:302) ~[kryo-4.0.2.jar:?]
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:731) ~[kryo-4.0.2.jar:?]
	at co.paralleluniverse.io.serialization.kryo.ReplaceableObjectKryo.readObject(ReplaceableObjectKryo.java:92) ~[quasar-core-0.7.10-jdk8.jar:0.7.10]
	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) ~[kryo-4.0.2.jar:?]
	... 21 more

Solution

The solution would be to register a custom serializer with Kryo to override its unsound java.util.Map serializer. However, I cannot find anything in the Corda documentation on how to do this.

Cause

The cause of the problem is that Kryo’s serializer is generalized in an unsound way to all implementations of java.util.Map. This doesn’t work for the persistent immutable maps we are using.

Kryo incorrectly assumes that if a class implements java.util.Map, then it can safely create an instance of it by passing null for all constructor parameters. In our case, the class is a MapAdapter which expects a non-null underlying instance of an immutable map implementation as a constructor parameter, and passing null here breaks the invariants of the class.

Steps to reproduce

1. Clone https://github.com/ulrikrasmussen/kryo-unsound-map-deserialization
2. Run the integration test. It succeeds, but the exception is shown in the log.

[r3jirabot]

Automatically created Jira issue: CORDA-3717

[ulrikrasmussen]

I think this issue can be closed now: https://docs.r3.com/en/platform/corda/4.6/open-source/cordapp-custom-serializers-checkpoints.html