OOM error with master/slaves setup (zeromq, windows)

Hi !

Describe the bug

An out of memory error occurs with ZeroMQ trying to allocate a crazy amount of memory in decoded_allocator, sometime up to several petabytes. This might very well be a ZeroMQ bug : OUT OF MEMORY (bundled\zeromq\src\decoder_allocators.cpp:89)

I added some logs and recompiled pyzmq to check what’s going on. Upon further investigation, _max_counters seems to take a crazy value at some point. See zmq_logs.txt As you can see, allocator instance 0x0000016A9270F700 is constructed with _max_counters=249, but before crash its value has changed to 1557249601288, which causes a malloc of several terabytes.

Steps to reproduce

Sorry, I couldn’t find a surefire way to reproduce this one. It seems kind of random. It sometime happens before the test is even started, sometime when the test is stopped. Sometime it doesn’t happen at all. It does seem to happen more often when stopping a test in the web UI. Simply run the ps1 attached and do some stuff in the web UI.

Environment

• OS: Windows 10.0.18362.778
• Python version: 3.6
• Locust version: 0.14.6
• Locust files : test_locust.zip

I managed to repro the bug on two computers : my work computer and my personal computer. Both are on Windows 10/Python 3.6 that comes with VS2017, but my personal computer has a pristine python environent, just ran pip install locustio.

Am I doing something I’m not supposed to ?

[Matthew--Townsend]

@RichardLions Good job updating the other bug (zeromq/pyzmq#1555) and finding a possible cause. Silly me, I thought the error (out of memory) could be something to do with running out of memory. 😃 I did try to run some python memory profilers but all I saw was a very flat memory allocation over time and nothing alarming.

[RichardLions]

pyzmq 22.2.1 has been released containing the fix for this issue.

https://github.com/zeromq/pyzmq/releases/tag/v22.2.1