[dask-on-ray] Memory efficiency: Dask arrays should always be passed by reference only

When passing a (persisted) dask array with np.ndarray chunks to some ray remote function, the whole dask array does get serialized. I created a notebook showing the consequence: https://gist.github.com/Hoeze/26430689bb3252ebb71b9ae08242d09a

Instead, the np.ndarray chunks should be put to the object store and the dask array should only contain references to the object store.

This could be solved by different ways, e.g.

• some ray_put_dask function that converts every chunk to a delayed ray.get() method
• change dask.persist() to return chunk references instead of np.ndarray-typed chunks

@rkooo567 @clarkzinzow

[clarkzinzow]

The core issue is that ray_dask_get won’t be able to tell if it’s being called by dask.compute() or dask.persist(), the call will look exactly the same to ray_dask_get in either case, which is why a user-supplied ray_persist=True flag would be necessary. And I agree that an explicit function would be better! I think that’s the best option for now.

[rkooo567]

We have an internal api to run the function on every worker. Maybe we should just support ray.util.setup_dask_on_ray() which sets the config on all workers + apply our custom shuffle optimization