Ray starts too many workers (and may crash) when using nested remote functions.

This is very similar to the earlier issue #231. One proposed solution was implemented by @stephanie-wang in https://github.com/ray-project/ray/pull/425.

Users sometimes encounter variants of the following bug and have no idea what is going wrong.

Running the following workload requires about 500 workers to be started (to execute all of the g tasks which are blocked in the call to ray.get before the f tasks start getting executed.

import ray
ray.init()

@ray.remote
def f():
    return 1

@ray.remote
def g():
    return sum(ray.get([f.remote() for _ in range(10)]))

ray.get([g.remote() for _ in range(500)])

Workarounds:

• Start fewer g tasks
• Divide g into two parts, e.g.,

  @ray.remote
  def g_part_a():
      return [f.remote() for _ in range(10)]
  
  @ray.remote
  def g_part_b(*results):
      return sum(results)
  
  intermediate = ray.get([g_part_a.remote() for _ in range(500)])
  ray.get([g_part_b.remote(*ids) for ids in intermediate])
  

• Use custom resources to constrain the number of g tasks running concurrently (suggested by @ericl).

Potential Solutions:

• Make the scheduler prioritize the f tasks over the g tasks (e.g., the strategy in #425 or some sort of LIFO policy.

In the meantime, we can easily detect that we’ve started way too many workers and push a warning to the user with a link to some possible workaround.

cc @stephanie-wang @ericl

[Lewisracing]

Hi is this issue improved in the latest ray verisions (e.g. 1.1.0)?

[robertnishihara]

@nmayhall-vt there is something a little subtle happening here.

If you are doing something like

x_id = ray.put(x)
y_id = ray.put(y)

@ray.remote
def process(data):
    x = ray.get(data[0])
    y = ray.get(data[1])

    # Use x and y

process.remote([x_id, y_id])

Then you should be able to avoid the issue by doing

x_id = ray.put(x)
y_id = ray.put(y)

@ray.remote
def process(x, y):
    # Use x and y

process.remote(x_id, y_id)

The reason is that when you call ray.get inside of a remote function, Ray will treat the task as “not using any resources” until ray.get returns, and so will potentially schedule additional tasks, which may require additional workers to be created.do this,

Does that make sense?

Also, is it giving the warning and then crashing? Or is it giving the warning and then succeeding?