question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[BUG] Without use_master_when_local=True minion ignores master parameter

See original GitHub issue

In this config:

file_client: local
master: 1.2.3.4
use_master_when_local: False

minion is tries to connect to 127.0.0.1:

21:19:31 - salt.minion:245 - DEBUG - Master URI: tcp://127.0.0.1:4506

But if I set use_master_when_local: True minion starts to connect to the master:

21:22:58 - salt.transport.zeromq:258 - DEBUG - Connecting the Minion to the Master URI (for the return server): tcp://1.2.3.4:4506
21:22:58 - salt.transport.zeromq:1300 - DEBUG - Trying to connect to: tcp://1.2.3.4:4506

Issue Analytics

  • State:open
  • Created 3 years ago
  • Comments:7 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
rrrixcommented, Sep 20, 2022

Just ran into this, but in an even worse way that killed my dev server 😦

With this config:

# Masterless Minion
master_type: disable
file_client: local

file_roots:
  base:
    - /srv/salt/state
pillar_roots:
  base:
    - /srv/salt/pillar

fileserver_backend:
  - roots

metadata_server_grains: True

And then I (accidentally) forgot to disable the salt-minion systemd service while using my masterless minion. My /var/log/salt/minion file quickly filled up my ~30GB root disk in about 2.5 hours.

# ls -lh /var/log/salt/minion
-rw-r----- 1 root root 29G Sep 20 18:44 /var/log/salt/minion

Repeated ad-infinitum:

2022-09-20 16:16:36,397 [salt.minion      :534 ][WARNING ][627] Master is set to disable, skipping connection
2022-09-20 16:16:36,397 [salt.minion      :1160][CRITICAL][627] Unexpected error while connecting to salt
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/minion.py", line 1134, in _connect_minion
    yield minion.connect_master(failed=failed)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1056, in run
    value = future.result()
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1070, in run
    yielded = self.gen.send(value)
  File "/usr/lib/python3/dist-packages/salt/minion.py", line 1365, in connect_master
    self.req_channel = salt.transport.client.AsyncReqChannel.factory(
  File "/usr/lib/python3/dist-packages/salt/transport/client.py", line 83, in factory
    return salt.channel.client.AsyncReqChannel.factory(opts, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/channel/client.py", line 127, in factory
    auth = salt.crypt.AsyncAuth(opts, io_loop=io_loop)
  File "/usr/lib/python3/dist-packages/salt/crypt.py", line 506, in __new__
    key = cls.__key(opts)
  File "/usr/lib/python3/dist-packages/salt/crypt.py", line 525, in __key
    opts["master_uri"],  # master ID
KeyError: 'master_uri'

~20 million exceptions between 2022-09-20 16:16:36,385 (first log entry) and 2022-09-20 18:44:12,741 (last log entry) - about 2.5 hours.

# grep 'opts\["master_uri"\]' /var/log/salt/minion | wc -l
20623786
0reactions
rrrixcommented, Sep 20, 2022

This works:

diff --git a/salt/minion.py b/salt/minion.py
index cecc4f4adf..40a1584f06 100644
--- a/salt/minion.py
+++ b/salt/minion.py
@@ -533,7 +533,7 @@ class MinionBase:
         if opts["master_type"] == "disable":
             log.warning("Master is set to disable, skipping connection")
             self.connected = False
-            raise salt.ext.tornado.gen.Return((None, None))
+            raise SaltSystemExit(1, "Master Connection Disabled")
 
         # Run masters discovery over SSDP. This may modify the whole configuration,
         # depending of the networking and sets of masters.

As far as I can figure, I don’t think master_type: disable was actually ever tested - at least not thoroughly. The previous line raise salt.ext.tornado.gen.Return((None, None)) has literally no effect, given that the function is always returned as a coroutine generator, which itself lives inside an infinite loop.

I’m looking a little bit more to see what test cases might look like for a PR, but I’m afraid I might have to build out a non-trivial amount of code to test if any Salt daemon should self-exit, given a certain configuration (such as this).

This patch also may not be viable for a scenario such as:

  • A master is configured
  • The minion has a valid connection (Approved keys) to the master
  • A user wants to use salt-call with a temporary configuration of master_type: disable, via a Saltfile for example, (instead of, for some reason, using --local)
Read more comments on GitHub >

github_iconTop Results From Across the Web

Incorrect Parameters (1-48) :: Minion Masters Bug Reporting
ErrorCode: Incorrect Parameters (1-48) - Same thing when I try both Mayhem battle and Mayhem team battle. I have the output log I...
Read more >
Feature #18351: Support anonymous rest and keyword rest ...
Anonymous rest and keyword rest arguments were already supported in method parameters, this just allows them to be used as arguments to other...
Read more >
This Minion Bug has been in the game for 10+ ... - YouTube
Caenen's Reddit Thread: https://www.reddit.com/r/leagueoflegends/comments/hyr8dn/red_side_cannon_minions_have_20_lower_attack/➤ Send me ...
Read more >
Storing Static Data in the Pillar - Salt Project Documentation
Here is an example using the ignore_missing keyword parameter to ignore ... all pillar data from servers pillar is delivered to minions and...
Read more >
Bug descriptions — spotbugs 4.7.3 documentation
Consider using the equals(Object) method instead. ES: Comparison of String parameter using == or != (ES_COMPARING_PARAMETER_STRING_WITH_EQ)¶. This code compares ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found