Support for multiple protocols or heterogenous protocols

I have a use case where the client<=>scheduler communication needs to be proxied, so we use the websocket protocol.

However, doing so means that scheduler<=>worker and worker<=>worker communication must also use websockets as we assume consistency.

This results in performance issues on systems with high-performance networking like Infiniband or NVLink. We usually use UCX for worker<=>worker communication to get the best out of the hardware.

I wanted to start a discussion around a solution for this. The two things that spring to mind are:

• Add support for multiple protocols simultaneously.
• Allow for one protocol for client<=>scheduler and a different one for worker<=>worker.

[mrocklin]

Yeah, I did this shortly after the websocket work, probably for the same reasons that you’re looking into it now.

As a heads-up, websockets work, but didn’t feel bulletproof.

[mrocklin]

I think there would be a need to specify which ports the different protocols listen on and set this from the dask-scheduler CLI.

You’e probably already figured this out by now, but many arguments now accept lists. Also, for complex options I’ve mostly switched to using dask-spec directly. It’s nice to cut out the CLI layer and just focus on the Scheduler class options.