LocalCUDACluster doesn't use multiple GPUs when cuDF is import

Only the first GPU is used when importing cudf. If I remove that import, all GPUs get assigned work to do.

from dask_cuda import LocalCUDACluster
from dask.distributed import Client
import numpy as np
import cupy
import dask
import dask.array as da
import cudf

if __name__ == '__main__':
    cluster = LocalCUDACluster()
    client = Client(cluster)

    x = cupy.random.random((100000, 1000))
    d = da.from_array(x, chunks=(10000, 1000), asarray=False)
    u, s, v = np.linalg.svd(d)
    s, v = dask.compute(s, v)

Obviously, this example is not really using cudf for anything, just serving for demonstration purposes.

The only workaround I found for this is to start dask-scheduler and dask-cuda-worker outside of my python script. Am I missing something or doing something wrong?

[pentschev]

@VibhuJawa you’re not importing cudf directly, but you are importing it indirectly when you import dask_cudf. I think this is where the context is coming from.

[pentschev]

Given that there has been extensive usage of LocalCUDACluster with cuDF in multiple GPUs recently, I believe this has been solved, closing.