Wrong `minion_id` in matcher methods.

Description of Issue

When filtering pillars with match.filter_by, they don’t get matched correctly. Reason is a wrong minion_id in the matcher methods. This bug was introduced in https://github.com/saltstack/salt/pull/48809/ , hence the question if that was somewhat intentional.

More specifically, it because of this deletion https://github.com/saltstack/salt/pull/48809/files#diff-889b4ad7294d8f45dfd1c3a87b9ba427L44

Can provide PR after clarification.

Setup

Any Setup with minion-filtered pillars, as described in https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.match.html

Steps to Reproduce Issue

Setup minion-filtered pillars as described in https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.match.html

Versions Report

Most of the 2019 versions, introduced with the PR https://github.com/saltstack/salt/pull/48809/

[peter-slovak]

We’re running into the same problem. Although getting minion_id from __opts__ should be a safe fallback from grains['id'], the reality is that __opts__['minion_id'] actually contains a different minion’s name.

That’s why the pcre_match keeps working and glob_match broke down - the first one uses opts['id'] (which in our case contains the correct minion name), the other one uses __opts__['minion_id'] (which contains another minion’s name).

I’m not sure how this is possible; my guess would be that __opts__['minion_id'] is somehow, somewhere modified as a result of a race or thread unsafety.

This problem goes away for a while after we restart the master. However, with 100% reproducibility, after we render the pillar on all minions (salt --async '*' pillar.items), the problem is back.

[peter-slovak]

FWIW, changing minion_id = opts.get('minion_id', opts['id']) to minion_id = opts['id'] in matchers/glob_match.py works around the problem (thus bypassing the incorrect value in opts['minion_id']).