Mocker Object Longevity

I am having flaky-testing trouble testing the pyserial module with pytest-mock. I think the mocker objects are living longer than they should (or I am misunderstanding/misusing pytest).

First off, I also was unable to use autospec on a pyserial object, which I investigated quite a bit but to no resolution. It was giving me the same error seen here pyside2-bug. So that, and my issue below, may be the same pyserial issue?

Onto my issue, I have cut down the code and made my own test cases to demonstrate the behavior I’m seeing. I believe that the serial object or my sr_mock object are not being destroyed between tests. I also think there may be other issues at play here because of the difference in failure modes of test-cases 2 & 3. Test-case 2 passes the call_count assert that catches test-case 3, but fails on the next call_count check. Both test-cases fail with different numbers showing up for call_counts on repeat calls.

I have been calling with “python -m pytest --count=10000” in the command line (windows 10). Each test-case shows about how many failures occur out of the 10000 runs, this is shown in the Testcase-Meanings section below.

Code to test:

#SerialReader.py

import serial

class SerialReader():
    def __init__(self):
        self.ser = serial.Serial()
        self.session_started = False
    def close_serial(self):
        self.ser.close()
        self.session_started = False

Test file: Adjust ‘testcase’ number to try different configurations

#test_SerialReader.py

import pytest
from pytest_mock import mocker
import SerialReader

## Define Testcase Number
testcase = 0


@pytest.fixture
def sr_mock(mocker):
    if testcase not in [1, 6]:
        mocker.patch('serial.Serial.close')
    if testcase not in [2, 5, 6]:
        mocker.patch('serial.Serial.__del__')
    sr_mock = SerialReader.SerialReader()
    return sr_mock

def test_main(sr_mock, mocker):
    if testcase not in [3, 5]:
        mocker.patch('serial.Serial.close')
    mocker.patch.object(sr_mock, 'session_started', new=True)
    assert sr_mock.ser.close.call_count == 0
    sr_mock.close_serial()
    assert sr_mock.ser.close.call_count == 1
    assert sr_mock.session_started == False

@pytest.mark.xfail
def test_mess_up(sr_mock, mocker):
    mocker.patch.object(sr_mock, 'session_started', new=True)
    with pytest.raises(AssertionError):
        assert sr_mock.session_started == False  # Identical "Mess-up Assert" inside pytest.raises
    if testcase not in [4, 5, 6]:
        assert sr_mock.session_started == False  # "Mess-up Assert"

Testcase Meanings:

Test Lines:
    fixture.close = sr_mock(mocker.patch('serial.Serial.close'))
        Overriden patch by main.close
    fixture.del = sr_mock(mocker.patch('serial.Serial.__del__'))
    main.close = test_main(mocker.patch('serial.Serial.close'))
        Duplicated patch overriding patch fixture.close
    messup.assert = test_mess_up(assert sr_mock.session_started == False)



Testcases Descriptions:   
0: All passing (0 in 10000 failures)
    fixture.close: active
    fixture.del: active
    main.close: active
    messup.assert: active

1: All passing
    fixture.close: disabled
        Shows this patch has no effect when overriden by main.close
    fixture.del: active
    main.close: active
    messup.assert: active

2: Intermittent Failures (~1504 in 10000 failures)
    fixture.close: active
    fixture.del: disabled
    main.close: active
    messup.assert: active

    Failure:
        test_main(assert sr_mock.ser.close.call_count == 1)
            call count returns numbers other than 1
            Passes previous check of (assert call_count == 0)
            Function then tries to call close once only inside sr_mock.close_serial()


3: Intermittent Failures (~119 in 10000 failures)
    fixture.close: active
    fixture.del: active
    main.close: disabled
        shows that fixture.close does not prevent errors, and this override is required
    messup.assert: active

    Failure:
        test_main(assert sr_mock.ser.close.call_count == 0)
            call count returns numbers other than zero
            neither fixture nor test has attempted to run close() yet

4: All passing (0 in 10000 failures)
    fixture.close: active
    fixture.del: active
    main.close: active
    messup.assert: disabled

5: All passing (0 in 10000 failures)
    fixture.close: active
    fixture.del: disabled
    main.close: disabled
    messup.assert: disabled
        Demonstrates with Test-4 that messup.assert causes test_main() to be flaky

6: All passing (0 in 10000 failures)
    fixture.close: disabled
    fixture.del: disabled
    main.close: active
    messup.assert: disabled
        Demonstrates with Test-5 that fixture.close has no effect on results
        Observed in testcase 1 as well

Environment:

platform win32 -- Python 3.7.3, pytest-5.2.0, py-1.8.0, pluggy-0.13.0 -- C:\Program Files\Python37\python.exe
cachedir: .pytest_cache
rootdir: .\pytestFail
plugins: cov-2.7.1, mock-1.11.0, randomly-3.1.0, repeat-0.8.0

[nicoddemus]

Hi @mokulus,

What’s happening is that when import nonexistent executes the first time, the Python import machinery will get the module object from sys.modules["nonexistent"], and place it in a global variable of the main module.

When the second test executes, the main module was already imported, which implies that the the main.py is not executed again (Python caches imports), so it still has the nonexistent object on its global namespace. This can be verified by calling importlib.reload(main) right after import main in the second test.

This is not really related to mock, but how the Python import machinery works.

@jacobian91 your issue is probably similar/related.

[nicoddemus]

@jacobian91 I think the only way to test this reliably is to execute your tests in a fresh Python interpreter.

You can use the testdir fixture to execute the test as a black box in a separate process:

pytest_plugins = 'pytester'

def test_add_one_to_random_once(testdir):
    testdir.makepyfile("""
        import sys

        def test_add_one_to_random_once(mocker):
            module = sys.modules['nonexistent'] = mocker.MagicMock()
            module.random.return_value = 1
            import main
            assert main.add_one_to_random() == 2
    """)

    result = testdir.runpytest_subprocess()
    result.stdout.fnmatch_lines(["* 1 passed *"])

Here we are creating a new temporary test file, executing pytest on it in a fresh subprocess, and then ensuring that the pytest output is what we expect.