Question: Is there an upper limit on number of workers?

I’m getting a thread.error when using a gather operation on a large number of nodes. Might be related to the environment I’m operating in. Here’s the stack trace:

Traceback (most recent call last):
  File "/home/venabled/pysrc/navfeatdb/bin/distributed_air_extract.py", line 137, in <module>
    kp_list = executor.gather(r)
  File "/home/venabled/anaconda2/lib/python2.7/site-packages/distributed/executor.py", line 745, in gather
    return sync(self.loop, self._gather, futures, errors=errors)
  File "/home/venabled/anaconda2/lib/python2.7/site-packages/distributed/utils.py", line 108, in sync
    raise result[0]
thread.error: can't start new thread

[toniatop]

It is surely late and probably not usefule, but the answer to your question is (in theory) yes. One could use: pbsdsh -v bash -l dask-worker scheduler to launch as many workers as cores available across all nodes

[mrocklin]

Is there a way to tell Torque/PBS to run dworker itself, rather than running a shell and then ssh-ing into each machine?