Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

failure to autoscale unless workers are already present

See original GitHub issue

I am testing the PBSCluster along with autoscaling. It seems that I am unable to get the cluster to launch any workers without explicitly starting at least one worker. I would expect that this configuration would scale from 0 to 10 (180 processes) without further interaction/configuration.

    cluster = PBSCluster(queue='default',
                         walltime='01:00:00',
                         project='MyAccount',
                         resource_spec='1:ncpus=36:mpiprocs=36:mem=109GB',
                         interface='ib0',
                         threads=4,
                         processes=18)
    client = Client(cluster)
    cluster.adapt(minimum=0, maximum=10)

@mrocklin - this may actually be a problem with the dask adaptive cluster but I wanted to discuss here to see if I am missing something obvious specific to PBS.

Issue Analytics

State:
Created 5 years ago
Comments:9 (9 by maintainers)

Top GitHub Comments

1reaction

mrocklincommented, Mar 29, 2018

Can you report the contents of cluster._adaptive.log ?

On Thu, Mar 29, 2018 at 3:35 PM, Joe Hamman notifications@github.com wrote:

I am testing the PBSCluster along with autoscaling. It seems that I am unable to get the cluster to launch any workers without explicitly starting at least one worker. I would expect that this configuration would scale from 0 to 10 (180 processes) without further interaction/configuration.
cluster = PBSCluster(queue='default',
                     walltime='01:00:00',
                     project='MyAccount',
                     resource_spec='1:ncpus=36:mpiprocs=36:mem=109GB',
                     interface='ib0',
                     threads=4,
                     processes=18)
client = Client(cluster)
cluster.adapt(minimum=0, maximum=10)
@mrocklin https://github.com/mrocklin - this may actually be a problem with the dask adaptive cluster but I wanted to discuss here to see if I am missing something obvious specific to PBS.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dask/dask-jobqueue/issues/26, or mute the thread https://github.com/notifications/unsubscribe-auth/AASszP7j_X4s7IepLV44BwOnbXqxShn6ks5tjTeVgaJpZM4TA3ut .

0reactions

jhammancommented, Jul 16, 2018

closed via #63