Tests: Collection of flukes on CI

Hi again,

while submitting some patches recently, we observed occasional flaky behavior of the test suite on CI. ^1. This issue intends to collect references to them, in order to find out about the root cause.

At ^2, we discovered test_plugin_twitter_tweet_attachments failed on PyPy 3.7 ^3 with #690. @caronc responded to this with:

This happens ALL the time; every now and then a random test that never failed (ever) in the past just fails on TravisCI. It’s usually always related to the Attachments. […] I can’t tell if it’s a Travis-CI issue, or the way my tests are written, or if if it’s some mulch-threading issue with Apprise and the timing of some of the attachment tests (causing it to pass 99.9% of the time).

Thank you for sharing your knowledge here, I will have another look at the test cases related to attachments when I can find some time.

If it happens again, even if I’m not in front of my PC, i can give easily give Travis CI runner a kick. Feel free to just ask me to do so… (or do a force-push on the branch again; that’ll trigger it to go again too). 🤷‍♂️

I think it will be fortunate to use the “force-push” variant, in order to keep references to the failed builds on Travis CI, in order to build up more knowledge where and how often this error happens. If you like that approach, feel free to add corresponding references to this issue while we go.

With kind regards, Andreas.

[caronc]

The previous runner was so slow, this test passed. 0.4995 is pretty close. I think the test just needs to be tweaked to allow for this. No worries here at all

[amotl]

PyPy3.9 again.

_______________________ test_plugin_fcm_oauth_no_colors ________________________

mock_post = <MagicMock name='post' id='66486064'>

    @pytest.mark.skipif(
        'cryptography' not in sys.modules, reason="Requires cryptography")
    def test_plugin_fcm_oauth_no_colors(mock_post):
        """
        Verify `color=no` work as intended.
        """
    
        obj = Apprise.instantiate(
            f'fcm://mock-project-id/device/?keyfile={FCM_KEYFILE}'
            '&color=no')
>       assert mock_post.call_count == 0
E       AssertionError: assert 1 == 0
E        +  where 1 = <MagicMock name='post' id='66486064'>.call_count

test/test_plugin_fcm.py:543: AssertionError

– https://github.com/caronc/apprise/actions/runs/3665208554/jobs/6196128736#step:10:125