Memory issue for rechunk

Hi,

I have a large numpy array (1000x1000x2000). When I tried to rechunk it into piecies in dask.ditributed, the memory will keep increasing and raise memory error eventually. My memory is 128 GB while the array is about 8 GB. So I am not clear what is going on here. My code is as follows

import numpy as np
import dask.array as da
from dask.distributed import Client

client = Client()
data = np.random.rand(1000,1000,2000).astype(np.float32) 
future = client.scatter(data)
daskdata = da.from_delayed(future,shape=data.shape,dtype=np.float32)
daskdata2 = daskdata.rechunk(300,300,900).persist()

The code above works perfectly fine for a smaller array, e.g. 1000x1000x1000. So I don’t understand what’s the reason here.

[mrocklin]

I suspect that your single large numpy array is larger than the intended memory for any of your individual processes. You might try one of the following options:

1. Use fewer processes with more memory. Consider using different options for LocalCluster other than the default.
2. Use just your single process with Client(processes=False)
3. Split your large numpy array into smaller pieces before you send it to the cluster.

Split locally first

x = da.from_array(x, chunks=...)
x = x.persist(get=dask.threaded.get)  # split locally
x = x.persist()  # persist to cluster

Use a cluster with a different process/thread breakdown

cluster = LocalCluster(n_workers=4, threads_per_worker=4)
client = Client(cluster)

Generally speaking Dask expects any particular value to be able to fit easily in the memory of one of your processes. If you use the defaults it creates as many processes as you have logical cores, which will break down your memory space.

[mrocklin]

Typically people don’t handle very large numpy arrays. They break them up somehow. Yes, HDF5 or NetCDF would work fine. You might also create your data in chunks. The right way to do things often depends on your application.