[QST] How to use multiple threads per GPU worker?

While running a large job, I noticed with watch -n 1 nvidia-smi that my GPUs were relatively underutilized.

I attempted to give each GPU worker more threads on which to process tasks simultaneously with threads_per_worker=2.:

from dask.distributed import Client, wait
from dask_cuda import LocalCUDACluster
import dask, dask_cudf

cluster = LocalCUDACluster(ip='0.0.0.0', threads_per_worker=2)
client = Client(cluster)
# print client info
client

The cluster starts up fine, and even begins processing tasks with twice as many streams, as expected. However, progress as reported by the Dask Dashboard locks up shortly afterwards on a DAG that completes successfully with the typical single-thread per worker.

nvidia-smi shows plenty of memory remaining memory per card, and Jupyter shows no errors or warnings.

Is this expected behavior? Any suggestions for how to diagnose the freeze?

[randerzander]

Maybe there are also other non-GPU tasks that the workers are spending time on.

Good point. The above tests were against gzipped files, so more processes reduced the overhead of host-side decompression.

When I switched to pre-decompressed data, some of the improvement dropped, but it is still significant, and variable only by about 10 seconds instead of 1 minute.

2 processes per worker, chunksize=‘512 MiB’: Wall time: 4min 46s

3 processes per worker, chunksize=‘512 MiB’: Wall time: 3min 55s

3 processes per worker, chunksize=‘1024 MiB’: Wall time: 3min 35s - 3min 46s

[jakirkham]

Should add Peter has been doing a lot of work adding support for PTDS. So that may be something worth trying out at some point