Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Slave count stuck at 3 instead of decreasing to 1 due to "missing"
See original GitHub issueMaster logs after slaves scaling down:
[2019-11-18 16:57:02,829] locust-opa-master-787476b668-74txj/INFO/locust.main: Starting web monitor at *:8089
[2019-11-18 16:57:02,830] locust-opa-master-787476b668-74txj/INFO/locust.main: Starting Locust 0.13.1
[2019-11-18 16:57:24,100] locust-opa-master-787476b668-74txj/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-cjkwb_594d3fd33f864c34aaabbb987cd79e65' reported as ready. Currently 1 clients ready to swarm.
[2019-11-18 16:57:24,210] locust-opa-master-787476b668-74txj/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-pljft_240da2aa2ada4a0993fad606fb5976e0' reported as ready. Currently 2 clients ready to swarm.
[2019-11-18 16:57:24,221] locust-opa-master-787476b668-74txj/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-zckdc_b42adb57f2d34ef885b0bcc228068102' reported as ready. Currently 3 clients ready to swarm.
[2019-11-18 16:57:24,253] locust-opa-master-787476b668-74txj/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-z2g29_7772633197e6471b9344951aba939c39' reported as ready. Currently 4 clients ready to swarm.
[2019-11-18 16:57:24,356] locust-opa-master-787476b668-74txj/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-d95g2_a3a40cf055d749ea8b6c0e3246f22faa' reported as ready. Currently 5 clients ready to swarm.
[2019-11-18 16:57:24,381] locust-opa-master-787476b668-74txj/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-mm7dt_ee79eead69c84802a096a12f6e6ef1e5' reported as ready. Currently 6 clients ready to swarm.
[2019-11-18 16:57:24,388] locust-opa-master-787476b668-74txj/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-b7c2k_a961e9574df048739d1057d70dff35bd' reported as ready. Currently 7 clients ready to swarm.
[2019-11-18 16:57:24,612] locust-opa-master-787476b668-74txj/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-kfjh2_c080829438e64f76a02090adaad1a0dc' reported as ready. Currently 8 clients ready to swarm.
[2019-11-18 16:57:24,626] locust-opa-master-787476b668-74txj/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-d2kg5_d5ee1c3c3dcb4762b87872edcdcb0e28' reported as ready. Currently 9 clients ready to swarm.
[2019-11-18 16:57:24,636] locust-opa-master-787476b668-74txj/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-hwh4v_77d2b07171a549fd9af73a8a86222c81' reported as ready. Currently 10 clients ready to swarm.
[2019-11-18 16:57:24,690] locust-opa-master-787476b668-74txj/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-vpfnf_5217f03d668f4bf2b3ae7580e8e701eb' reported as ready. Currently 11 clients ready to swarm.
[2019-11-18 16:57:24,764] locust-opa-master-787476b668-74txj/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-sl24v_0da998e92bcd46b1ba7161d260bc3edf' reported as ready. Currently 12 clients ready to swarm.
[2019-11-18 16:57:24,786] locust-opa-master-787476b668-74txj/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-nw5ln_97ac9424368546b1b947441ce2d11f3a' reported as ready. Currently 13 clients ready to swarm.
[2019-11-18 16:57:24,804] locust-opa-master-787476b668-74txj/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-w6lhx_fa3a32665a814af8af8fca9657814d1f' reported as ready. Currently 14 clients ready to swarm.
[2019-11-18 16:57:24,932] locust-opa-master-787476b668-74txj/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-qr4tf_8ff013c045e7445494154d52f3a5dda7' reported as ready. Currently 15 clients ready to swarm.
[2019-11-18 16:57:25,074] locust-opa-master-787476b668-74txj/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-jb2rr_8e423e848d9b442e9cfc02d6c1cd6aca' reported as ready. Currently 16 clients ready to swarm.
[2019-11-18 16:57:34,843] locust-opa-master-787476b668-74txj/INFO/locust.runners: Slave locust-opa-slave-86cf6df9fb-zckdc_b42adb57f2d34ef885b0bcc228068102 failed to send heartbeat, setting state to missing.
[2019-11-18 16:57:34,843] locust-opa-master-787476b668-74txj/INFO/locust.runners: Slave locust-opa-slave-86cf6df9fb-z2g29_7772633197e6471b9344951aba939c39 failed to send heartbeat, setting state to missing.
[2019-11-18 17:02:30,550] locust-opa-master-787476b668-74txj/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-b7c2k_a961e9574df048739d1057d70dff35bd' quit. Currently 13 clients connected.
[2019-11-18 17:02:30,557] locust-opa-master-787476b668-74txj/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-vpfnf_5217f03d668f4bf2b3ae7580e8e701eb' quit. Currently 12 clients connected.
[2019-11-18 17:02:30,559] locust-opa-master-787476b668-74txj/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-sl24v_0da998e92bcd46b1ba7161d260bc3edf' quit. Currently 11 clients connected.
[2019-11-18 17:02:30,560] locust-opa-master-787476b668-74txj/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-jb2rr_8e423e848d9b442e9cfc02d6c1cd6aca' quit. Currently 10 clients connected.
[2019-11-18 17:02:30,563] locust-opa-master-787476b668-74txj/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-kfjh2_c080829438e64f76a02090adaad1a0dc' quit. Currently 9 clients connected.
[2019-11-18 17:02:30,569] locust-opa-master-787476b668-74txj/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-hwh4v_77d2b07171a549fd9af73a8a86222c81' quit. Currently 8 clients connected.
[2019-11-18 17:02:30,571] locust-opa-master-787476b668-74txj/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-w6lhx_fa3a32665a814af8af8fca9657814d1f' quit. Currently 7 clients connected.
[2019-11-18 17:02:30,580] locust-opa-master-787476b668-74txj/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-nw5ln_97ac9424368546b1b947441ce2d11f3a' quit. Currently 6 clients connected.
[2019-11-18 17:02:30,581] locust-opa-master-787476b668-74txj/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-pljft_240da2aa2ada4a0993fad606fb5976e0' quit. Currently 5 clients connected.
[2019-11-18 17:02:30,588] locust-opa-master-787476b668-74txj/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-mm7dt_ee79eead69c84802a096a12f6e6ef1e5' quit. Currently 4 clients connected.
[2019-11-18 17:02:30,599] locust-opa-master-787476b668-74txj/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-cjkwb_594d3fd33f864c34aaabbb987cd79e65' quit. Currently 3 clients connected.
[2019-11-18 17:02:30,599] locust-opa-master-787476b668-74txj/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-d2kg5_d5ee1c3c3dcb4762b87872edcdcb0e28' quit. Currently 2 clients connected.
[2019-11-18 17:02:30,604] locust-opa-master-787476b668-74txj/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-d95g2_a3a40cf055d749ea8b6c0e3246f22faa' quit. Currently 1 clients connected.
The response from /stats/requests endpoint looks like this for very long time:
{
"current_response_time_percentile_50": null,
"current_response_time_percentile_95": null,
"errors": [],
"fail_ratio": 0.0,
"slaves": [
{
"id": "locust-opa-slave-86cf6df9fb-zckdc_b42adb57f2d34ef885b0bcc228068102",
"state": "missing",
"user_count": 0
},
{
"id": "locust-opa-slave-86cf6df9fb-z2g29_7772633197e6471b9344951aba939c39",
"state": "missing",
"user_count": 0
},
{
"id": "locust-opa-slave-86cf6df9fb-qr4tf_8ff013c045e7445494154d52f3a5dda7",
"state": "ready",
"user_count": 0
}
],
"state": "ready",
...
I think the master is not removing the missing slaves?
Issue Analytics
- State:
- Created 4 years ago
- Comments:9 (6 by maintainers)
Top Results From Across the Web
When Did Slavery Really End in the North? - Civil Discourse
The perception about the United States in the period before the Civil War is that the North was “free” and the South was...
Read more >Narrative of the Life of Frederick Douglass, an American Slave ...
The experience of FREDERICK DOUGLASS, as a slave, was not a peculiar one; ... for the slave population; and any amount of cruelty...
Read more >The Dark Side of Thomas Jefferson - Smithsonian Magazine
Jefferson's 4 percent theorem threatens the comforting notion that he had no real awareness of what he was doing, that he was “stuck”...
Read more >The Slave Trade | National Archives
Related Primary Sources · Sherman, Isaac v. Charming Sally, Schooner, 1803 · The Act Prohibiting the Importation of Slaves, 1808 · U.S. vs...
Read more >The Economics of the Civil War - EH.net
Figure 1 plots the total value of all slaves in the United States from 1805 to ... be reduced substantially if instead of...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
I’ll reopen when I can reproduce it.
Hi, I increased my POD count to compensate (only a workaround…) I don’t have time yet to see why Workers are beginning to be “missed” as soon I figure something out I’ll let you know.