Refactor callbacks

Like structs, native-to-Java callbacks look out-of-place in Java without special JVM support. Now that LWJGL 3 requires Java 8, we should rethink their design and implementation.

Current design

• Callback types are abstract classes and they extend a base abstract class (Callback.<T>).
• Callback types are represented by the corresponding abstract class in bindings (function parameters and struct members).
• SAM conversion is enabled with factory methods in the callback classes. There’s a SAM interface in each class and a public static <CallbackClass> create(SAM sam) method.
• A JNI weak global reference is created that points to the callback instance and stored in the native callback object (as “user data”).
• We check for premature GC when dereferencing the JNI weak global reference and display an appropriate error, without crashing.
• Callback instances must be freed, by calling the .free() method on the instance.

Pros

• Has worked without issues so far.
• Straightforward clean-up.
• Users are forced to store a strong reference to the callback instance. They don’t have to RTFM, they figure it out on their own when callbacks start failing after the first GC. This then makes it relatively obvious that clean-up must take place.

Cons

• SAM conversion does not work with abstract classes, only with functional interfaces.
• There’s both a class and an interface (SAM) for each callback type.
• Callback class names are arbitrary; they are not present as a type in the native API.
• The user is forced to store a strong reference to the callback instance.

Recent changes

• LWJGL 3 now requires Java 8. We can use static and default methods in interfaces.
• libffi has been replaced by dyncall. One important difference with dyncall is that the thunk pointer and the callback object handle are guaranteed to be one and the same. This means that we only need a single pointer to reference the entire (native and Java) callback structure.

Proposed design

• Each callback type is a functional interface.
• Callback argument decoding can happen directly in the interface, with a default method.
• The Callback class becomes a utility class that only creates and destroys callback objects.
• A callback object is a plain long value (opaque handle). Callback function arguments and member fields become long.
• There are method overloads that take a functional interface, create a callback object from it and pass it to the native function/struct.
• Callback type instances are stored in strong JNI global references.
  • Note that non-capturing lambdas and method references are never GCed anyway.
• The user may or may not store the callback object in Java code. In many bindings they don’t have to store it, it’s readily available via the API (e.g. glfwSet<Type>Callback function return the previously set callbacks).

Pros

• There’s a single functional interface per callback type.
• One less indirection (no abstract class -> SAM delegation).
• Interface names won’t show up in code that uses lambdas or method references.
• Callback object handles stored in user code will be simple long, not arbitrary types.
• Less callback objects stored in user code.
• No worries about premature GC.

Cons

• Memory leaks if the user doesn’t perform clean-up. This is generally of low-concern, most applications will use a low-number of callbacks without much (if any) state.
• Non-obvious clean-up:
  • Must retrieve callback object handle using the appropriate binding API (e.g. glGetPointer(GL_DEBUG_CALLBACK_FUNCTION)), if the handle was not already stored in user code.
  • Must use Callback.free(handle) (vs cbInstance.free() currently).

Example 1, similar to current design:

private long keyCB; // callback handle is a long instead of an object
// ...
// callback setup, must pass lambda or method reference to the static create method
glfwSetKeyCallback(window, keyCB = GLFWKeyCallback.create((windowHandle, key, scancode, action, mods) -> {
   // ...
 }));
// cleanup
Callback.free(keyCB);

Example 2, without storing the callback handle:

// callback setup, lambda or method reference passed directly to the API
glfwSetKeyCallback(window, (windowHandle, key, scancode, action, mods) -> {
   // ...
 });
// cleanup
Callback.free(glfwSetKeyCallback(window, null));

[Spasi]

Sorry, I’m probably not explaining things clearly. Lets try again.

The dyncall callback object (DCCallback*) is a struct. The long value you get from <CallbackInterface>.create(...) is an opaque pointer to that struct. The first member of that struct is the thunk, which is basically the function pointer passed to a native API that expects a callback pointer. Since it’s the first member in the struct, the DCCallback* pointer is also the function pointer. You have both using a single long value.

The only thing missing now is the Java callback (the instance that implements the callback interface). This is also stored in the callback object struct, in the userdata field. Since the struct is opaque, dyncall has a function for retrieving the userdata field: dcbGetUserData(DCCallback*). So you can do this:

long keyCB = GLFWKeyCallback.create((windowHandle, key, scancode, action, mods) -> {
   // ...
 });

GLFWKeyCallback javaCallback = memGlobalRefToObject(dcbGetUserData(keyCB));
// or simply
GLFWKeyCallback javaCallback = Callback.get(keyCB);

[octylFractal]

OK, I’ll do that instead. Thanks!