AsynchronousTlsChannelGroup#processPendingInterests can throw CancelledKeyException

In tests of the MongoDB Java driver that use this library, I’ve seen occasional, non-deterministic failures where AsynchronousTlsChannelGroup#processPendingInterests throws CancelledKeyException, causing AsynchronousTlsChannelGroup.loop to exit. It happens in cases where we are forcing the server to close the socket in order to test failure scenarios.

I’m not exactly sure why this is happening, but I do see that in AsynchronousTlsChannelGroup.loop there is already code that wraps calls to java.nio.channels.SelectionKey#interestOps(int) in a try/catch of CancelledKeyException. Does it make sense to do a similar thing in AsynchronousTlsChannelGroup#processPendingInterests , e.g.

  private void processPendingInterests() {
    for (SelectionKey key : selector.keys()) {
      RegisteredSocket socket = (RegisteredSocket) key.attachment();
      int pending = socket.pendingOps.getAndSet(0);
      if (pending != 0) {
        try {
          key.interestOps(key.interestOps() | pending);
        } catch (CancelledKeyException e) {
          // ignore
        }
      }
    }
  }

[stIncMale]

@martinandersson Below is my explanation of the problem and the fix.

The method AsynchronousTlsChannelGroup.RegisteredSocket.close may be called by any thread as a result of it calling AsynchronousTlsChannel.close. AsynchronousTlsChannelGroup.RegisteredSocket.close calls SelectionKey.cancel, which

Requests that the registration of this key’s channel with its selector be cancelled. Upon return the key will be invalid and will have been added to its selector’s cancelled-key set. The key will be removed from all of the selector’s key sets during the next selection operation.

If we now look at Selector, we can see that

The cancelled-key set is the set of keys that have been cancelled but whose channels have not yet been deregistered. This set is not directly accessible. The cancelled-key set is always a subset of the key set.

This means that selector.keys in AsynchronousTlsChannelGroup.processPendingInterests may return cancelled SelectionKeys. Calling key.interestOps on such SelectionKeys results in CancelledKeyException as per the documentation of SelectionKey.interestOps.

Thus, depending on how AsynchronousTlsChannelGroup and AsynchronousTlsChannel are used in a program, the program may have a race condition.

Two approaches are possible:

1. change the usage of SelectionKey.cancel, Selector.select, Selector.keys, SelectionKey.isValid, SelectionKey.interestOps methods in AsynchronousTlsChannelGroup in such a way that there can be no such race condition anymore;
2. catch and ignore CancelledKeyException when it happens as a result of a program having the race condition.

The second approach seems (maybe surprisingly) more optimal in this case because it is both simpler and introduces smaller performance overhead assuming that CancelledKeyException is thrown much more rarely than the method SelectionKey.cancel is called.

[marianobarrios]

The Selector API is already racy here. But a lot of “closing workflows” are racy and benign, as typically not much happens after a close to matter anyway.

Something that would help: having a test that show non-deterministic behavior due to this race.