[job submission] Support running supervisor actors in worker nodes

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

In some real world use cases, we will deploy the dashboard in a separate node which doesn’t include a Raylet process. We do this because we want to achieve high available of dashboard. But now in job submission, we use ray.init in dashboard to ensure the supervisor actor could be launched. The assumption is that dashboard and Raylet are collocated in one node.

So, can we use Ray Client in dashboard?

Use case

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[SongGuyang]

Seems we still have more than half a month before June. I will lead discuss first in Ant next week and go back to sync with you.

[edoakes]

I think this is backwards: the ray client server uses a hacky implementation to create the driver process that circumvents the standard process scheduling & runtime_env ref counting path. Instead it should do the same thing that the job submission server does: schedule a regular actor. This would unify how we do process management and environment setup across the board.

To solve the issue of ray.initing to the raylet, maybe we should have the actor calls happen in the per-node agent instead of the main dashboard process?