Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Support multiple workers per VM

See original GitHub issue

It may be more efficient to run many workers on larger VMs as opposed to larger numbers of worker processes on small VMs, presumably to avoid inter-VM communication. This was a suggestion that resulted from this thread on optimizing some dask array workflows: https://github.com/pystatgen/sgkit/issues/390.

@quasiben mentioned that threads per worker can be controlled with worker_options={"nthreads": 2 }, but there appears to be no way to run more than one worker on a single VM.

Issue Analytics

State:
Created 3 years ago
Reactions:1
Comments:33 (16 by maintainers)

Top GitHub Comments

2reactions

ansar-sacommented, Dec 2, 2021

Could we please request / create issue in distributed upstream to integrate this work.

For my use case it makes more sense to have a few very large machines than lots of small VMs.

1reaction

jacobtomlinsoncommented, Nov 4, 2022

@kumarprabhu1988 I think the cls need to be a string, and it probably needs to be "distributed.nanny.Nanny".

Top Results From Across the Web

VM deployed into CloudHub with Multiple workers

This app is deployed on 3 multiple workers on CloudHub. My questions is how does VM work with multiple CloudHub workers?? does it...

Multiple worker roles on the same instance - Stack Overflow

Web and Worker Roles are Window Server VMs, the main difference being Web Roles have IIS running. You may have multiple instances of...

Multi-worker training with Keras | TensorFlow Core

This tutorial demonstrates how to perform multi-worker distributed training with a Keras model and the Model.fit API using the tf.distribute.

Session host virtual machine sizing guidelines - Microsoft Learn

For scenarios with 20 or more users connected to a single VM, several smaller VMs would perform better than one or two large...

General-purpose machine family | Compute Engine ...

Shared-core machine types offer bursting capabilities that allow instances to use additional physical CPU for short periods of time. Bursting happens ...