What is the proper way to submit a job to the cluster?

If I have a local cluster, brought up with ray up command, what is the proper way to start e.g. impala job on the cluster?

1. ssh into head node and run python my_script.py or
2. ray submit cluster.yaml --tmux --start --stop my_script.py?

It is not really clear wether running a python script on any note with ray.init(redis_address) is sufficient to distribute the load across the cluster. For some reason my head node always takes all of the work, even though my cluster has 2 workers.

For example, the doc says (https://ray.readthedocs.io/en/latest/using-ray-on-a-cluster.html) that you can run code onthe cluster as follows:

import ray
ray.init(redis_address="<redis-address>")

import time

@ray.remote
def f():
    time.sleep(0.01)
    return ray.services.get_node_ip_address()

# Get a list of the IP addresses of the nodes that have joined the cluster.
set(ray.get([f.remote() for _ in range(1000)]))

Which works. But how do I apply this to RLLib library with impala or appo algorithms?

[richardliaw]

If you’re using rllib train, you can pass in --redis-address.

If you’re using a python script and you want to use the entire cluster, rllib never autodetects your cluster (because you can have multiple ray clusters at a time). You append ray.init(redis_address="<redis_address>") to your script as @kivo360 mentioned.

ray submit [script] is just a wrapper around ssh user@ipadress python [script].

Hope that helps.

[worldveil]

@edoakes should we post here references to ray job submit now that this is preferred way of doing things?

this is a top result on google now for “ray job submit python”