Failing tests

What problem has occurred? What issues has it caused?

Several of the recent tests are failures, possibly due to the recent refactoring #6948

For example, https://github.com/Charcoal-SE/SmokeDetector/runs/6320800336?check_suite_focus=true failed with the error traceback below.

What would you like to happen/not happen?

Tests should succeed when nothing is wrong (-:

May be relevant: When I run the tests locally, I get this warning:

test/test_blacklists.py:215
  /home/tripleee/charcoal/SmokeDetector/test/test_blacklists.py:215: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/mark.html
    @pytest.mark.xdist_group(name="long_runner_1")

Here is the traceback from the example:

def test_metasmoke():
        orig_tell_rooms_with = chatcommunicate.tell_rooms_with
        chatcommunicate.tell_rooms_with = dummy_tell_rooms_with
        msg = Fake({
            "owner": {
                "name": "ArtOfCode",
                "id": 121520,
                "is_moderator": False
            },
            "room": {
                "id": 11540,
                "_client": {
                    "host": "stackexchange.com"
                }
            },
            "_client": {
                "host": "stackexchange.com"
            }
        })
        msg_source = "metasmoke is {}. Current failure count: {} " + "({id})".format(id=GlobalVars.location)
    
        chatcommands.metasmoke(original_msg=msg, alias_used="ms-up")
        assert GlobalVars.MSStatus.is_up()
        chatcommands.metasmoke(original_msg=msg, alias_used="ms-down")
>       assert GlobalVars.MSStatus.is_down()
E       AssertionError: assert False
E        +  where False = <function GlobalVars.MSStatus.is_down at 0x7f41f81237f0>()
E        +    where <function GlobalVars.MSStatus.is_down at 0x7f41f81237f0> = <class 'globalvars.GlobalVars.MSStatus'>.is_down
E        +      where <class 'globalvars.GlobalVars.MSStatus'> = GlobalVars.MSStatus

test/test_chatcommands.py:656: AssertionError
----------------------------- Captured stderr call -----------------------------
[09:55:04.7[34](https://github.com/Charcoal-SE/SmokeDetector/runs/6320800336?check_suite_focus=true#step:10:34)] KeyError: ('stackexchange.com', 11540)
2022-05-06 09:55:04.73[36](https://github.com/Charcoal-SE/SmokeDetector/runs/6320800336?check_suite_focus=true#step:10:36)09 UTC
  File "/home/runner/work/SmokeDetector/SmokeDetector/chatcommunicate.py", line 482, in __call__
    result = self.__func__(*processed_args, alias_used=alias_used)
  File "/home/runner/work/SmokeDetector/SmokeDetector/chatcommands.py", line 926, in metasmoke
    if not is_privileged(msg.owner, msg.room):
  File "/home/runner/work/SmokeDetector/SmokeDetector/chatcommunicate.py", line 432, in is_privileged
    return user.id in _privileges[(room._client.host, room.id)] or user.is_moderator



  File "/home/runner/work/SmokeDetector/SmokeDetector/chatcommunicate.py", line 482, in __call__
    result = self.__func__(*processed_args, alias_used=alias_used)
  File "/home/runner/work/SmokeDetector/SmokeDetector/chatcommands.py", line 926, in metasmoke
    if not is_privileged(msg.owner, msg.room):
  File "/home/runner/work/SmokeDetector/SmokeDetector/chatcommunicate.py", line 432, in is_privileged
    return user.id in _privileges[(room._client.host, room.id)] or user.is_moderator

[09:55:04.738] KeyError: ('stackexchange.com', 11540)
2022-05-06 09:55:04.7[37](https://github.com/Charcoal-SE/SmokeDetector/runs/6320800336?check_suite_focus=true#step:10:37)917 UTC
  File "/home/runner/work/SmokeDetector/SmokeDetector/chatcommunicate.py", line 482, in __call__
    result = self.__func__(*processed_args, alias_used=alias_used)
  File "/home/runner/work/SmokeDetector/SmokeDetector/chatcommands.py", line 926, in metasmoke
    if not is_privileged(msg.owner, msg.room):
  File "/home/runner/work/SmokeDetector/SmokeDetector/chatcommunicate.py", line [43](https://github.com/Charcoal-SE/SmokeDetector/runs/6320800336?check_suite_focus=true#step:10:43)2, in is_privileged
    return user.id in _privileges[(room._client.host, room.id)] or user.is_moderator



  File "/home/runner/work/SmokeDetector/SmokeDetector/chatcommunicate.py", line [48](https://github.com/Charcoal-SE/SmokeDetector/runs/6320800336?check_suite_focus=true#step:10:48)2, in __call__
    result = self.__func__(*processed_args, alias_used=alias_used)
  File "/home/runner/work/SmokeDetector/SmokeDetector/chatcommands.py", line 926, in metasmoke
    if not is_privileged(msg.owner, msg.room):
  File "/home/runner/work/SmokeDetector/SmokeDetector/chatcommunicate.py", line 432, in is_privileged
    return user.id in _privileges[(room._client.host, room.id)] or user.is_moderator

[makyen]

As to the more prevalent sporadic errors (at least 3 today: 1, 2, 3:

FAILED test/test_chatcommunicate.py::test_validate_yaml - Failed: Privileged user 22202197 does not have a corresponding entry in users.yml

which I mention above, I’ve been working on adding threading locks on quite a bit of stuff which we use across threads, both in CI testing, but more importantly in normal operation of SD. Some of those changes should address the above issue, or at least reduce it. Given that those changes are adding a bunch of locks, I’d prefer to wait for the weekend to put them in. I’ll try to get what I have so far into a PR in the next day or two. While I’ve made a lot of changes, there’s still substantial sections of code which need to be reviewed for thread safety.

In addition to the changes I have so far for our code, I also have changes for the regex package. I still need to look in detail at both the ChatExchange and phonenumbers packages wrt. thread safety (a cursory check makes it appear both have issues which will need to be resolved).

[makyen]

It looks like this CircleCI test run confirms that the contents of the users.yml file which are read in the test/test_chatcommunicate.py::test_validate_yaml() test are different than what was on the disk prior to running pytest. So, either that test is somehow reading the contents of the file incorrectly, or we have something that’s changing the content that’s on the disk (which seems more likely). Given that we do have tests which do git operations, we should take a look at those to see if any of them change the contents of the files. It may mean we need to run those tests separately, or in some way prevent interference between tests.