Multiple cores per process/thread

We have a use case where we would like to dask.distributed to parallelize a python-bound C++ program that would work best if it could consume 8-32 threads depending on the problem size and will manage threading internally. Normally, a Dask worker is run on a node that has 32 cores with the worker using 2 processes at 1 thread each so that we can give each process 16 cores.

Looking at the code currently, it seems that nthreads = ncores / nprocesses without exceptions, is there a canonical way to change this so that we can orchestrate our normal Dask worker operation with dask-jobqueue?

[dgasmith]

This thread has moved through several design concepts, let me try to recap:

• Staying with the dask-worker language and defaults is likely optimal:

  --nthreads INTEGER     Number of threads per process. Defaults to number of
                         cores
  --nprocs INTEGER       Number of worker processes to launch.  Defaults to one.

• Resources per task should be handled through the resources client/worker commands as jobqueue/dask-worker does not have much of an understanding of a task.
• A proposed PR would simply allow the full specification of dask-worker parameters.
• Any PR should keep functionality where worker nthreads = ncores / nprocs by default.

We can approach this in a few ways:

• worker_nthreads = None or worker_threads_per_process = None which sets worker nthreads and default behavior if None.
• @louisabraham suggestion of having kwargs to exactly mimic dask-worker with nthreads/nprocs (would require deprecating the current processes keyword)
• one_thread_per_process = False/True, False is current behaviour while True sets nthreads=1.

I would very much vote for the nthreads/nprocs suggestion even if another warning needs to be thrown. This will keep the code logic and user understanding exactly the same as launching a dask-worker.

[jchodera]

I still would encourage API access to all relevant bits of your canonical dask-worker along with keeping the same name and meaning for these options, this would include nthreads.

This would indeed be great!