dask.distributed zarr reads slows down with additional workers

Problem description

I have a zarr store with around 50 arrays on disk with a wide variety of dtypes. all stored with blosc compressor. While trying to parallelize some calculations with dask, I noticed that adding more workers to local dask cluster slows down the _decode_chunk calls. In the documentation, there is some mention of blosc multithreading not working with multiprocessing and I tried setting BLOSC_NOBLOCK environment variable to see if makes a difference but didn’t see any difference. Is this expected behavior with blosc?

def test():
     zs = zarr.open('/tmp/zarr')
    for n,a in zs.arrays():
         a[:,:100].shape

delayed_res = []
for i in range(1000):
     delayed_res.append(dask.delayed(test))

dask,.compute(*dask_delayed)

Running this with a single worker each test call takes aroun 3.5 seconds, if I increase worker count to 10 than each call takes 4.5 seconds.

Version and installation information

• zarr.version : 2.2.0
• numcodecs.version: 0.5.5
• Python 3.6.6
• Linux
• zarr installed using conda

[alimanfoo]

FWIW in my experience I get best CPU utilisation when blosc is using 1 thread internally and I max out on number of dask workers (= cpu count).

[jakirkham]

Thanks. Let us know if other issues crop up.