Support pluggable EventLoop task queue

The default task queue used by EPoll and NIO EventLoop implementations is provided by JCTools, and while it is efficient it has some undesirable properties. Specifically, it is not non-blocking, and can lead to the EventLoop busy-spinning waiting for a task that may not be provided promptly because the offering thread’s execution has been suspended by the operating system. This is a rare scenario, but it will happen and some application maintainers might like to avoid it.

Unfortunately, there is no simple fix in the use of the queue; the relaxedPoll() method appears to be a candidate, but does not provide volatile visibility guarantees to a preceding offer(), meaning a wakeup could be missed (although we could write to a dummy volatile variable after offering to provide this). A queue identical to the JCTools one but using a putVolatile to set the item in the queue’s backing array would suffice to avoid the busy-spin blocking of the EventLoop, but the issue of future tasks being unreachable until the thread completes would remain.

Ideally, the queue used by these event loops would be pluggable, so that users can choose their ideal point on the spectrum of tradeoffs.

A simple replacement that guarantees progress is the JDK ConcurrentLinkedQueue, but at the cost of slightly more expensive offer() and poll(); the above mentioned modification to the JCTools offer() could be mixed with relaxedPoll() to simply avoid busy-spin blocking, and a custom Mpsc queue implementation could guarantee forward progress with only marginally higher cost. Permitting the application maintainer to pick their poison would be tremendously helpful.

[nitsanw]

@belliottsmith @franz1981 reading through the discussion and the java doc I realize JCTools should do a better job documenting the progress guarantees on this and other queues. I’ve filed a bug: https://github.com/JCTools/JCTools/issues/259

[belliottsmith]

If this is the issue under discussion, I’m not sure what the following suggestion means: “A queue identical to the JCTools one but using a putVolatile to set the item in the queue’s backing array would suffice to avoid the busy-spin blocking of the EventLoop”

It means the eventLoop (by using relaxedPoll()) would at least be able to proceed with other work such as reading/writing to a socket, or could stop if it had no other work to do, freeing up a thread for the producer to complete its work.

Obviously a preferable solution is to use a truly non-blocking queue, but blocking the eventLoop is worse than just blocking the external work queue.