Users are not distributed equally across workers.

Describe the bug

Running locust in k8s across hundreds of workers, I peeked the workers UI tab and wondered why do some workers report to run several users, when most of the cluster is still not running a single user.

Expected behavior

I expected the users to be equally distributed across my workers

Actual behavior

I ran this test with 600 workers

Steps to reproduce

I guess it reproduces with a high number of workers?

Environment

• OS: k8s
• Python version: locust latest official docker image
• Locust version: locust latest official docker image
• Locust command line that you ran: args: ["-f", "/loadtest/locustfile.py", "--worker", "--master-host", "locust-master"]
• Locust file contents (anonymized if necessary):

from locust import HttpUser, task
import subprocess

class LoadTestUser(HttpUser):
    @task
    def do_test(self):
        urlfile = subprocess.check_output(["bash", "-c", "find /loadtest/urlfiles -type f | shuf | head -n 1"]).rstrip()
        with open(urlfile) as f:
            urls = [l.rstrip() for l in f.readlines()]
        for u in urls:
            self.client.get(u, verify=False)

Basically the test fetches a random url list for a single flow in our system, and probes all the URLs.

[guysv]

OK setting my cluster with pod affinity so every locust worker get a core and the master get a private node did the trick.

Closing the issue for now, although it looks like running workers in big numbers requires some tricks to the heartbeat mechanism.

[cyberw]

So obviously the master is conjested

well, I wouldnt be so sure, but it is very possible.