test_configuration interferes with other unit tests

Environment

• pip version: dev
• Python version: 3.7
• OS: Fedora Linux
• Timezone: UTC+10

Encountered while working out why a full test run was failing for me locally while the CI on #6210 was passing.

Description

There seem to be a few cases where tests are passing when run in isolation, but failing when run in combination with other tests.

Expected behavior

• tox -e py37 passes without the splitting up into distinct groups used in the CI config

How to Reproduce

Cases that are failing for me in a full test run, but pass standalone:

• tests/unit/test_logging.py
• tests/unit/test_base_command.py

The main culprit appears to be tests/unit/test_configuration.py:

• tox -e py37 -- tests/unit/test_configuration.py tests/unit/test_base_command.py
• tox -e py37 -- tests/unit/test_configuration.py tests/unit/test_logging.py

Output

The test_configuration + test_logging failure looks like the latter is using my timezone rather than UTC:

py37 runtests: commands[0] | pytest --timeout 300 tests/unit/test_configuration.py tests/unit/test_logging.py
========================================================================================= test session starts ==========================================================================================
platform linux -- Python 3.7.2, pytest-3.8.2, py-1.7.0, pluggy-0.8.1
rootdir: /home/ncoghlan/devel/pip, inifile: setup.cfg
plugins: xdist-1.26.1, timeout-1.3.3, rerunfailures-6.0, forked-1.0.1, cov-2.6.1
timeout: 300.0s
timeout method: signal
timeout func_only: False
collected 27 items                                                                                                                                                                                     

tests/unit/test_configuration.py ......................                                                                                                                                          [ 81%]
tests/unit/test_logging.py .F...                                                                                                                                                                 [100%]

=============================================================================================== FAILURES ===============================================================================================
__________________________________________________________________________ TestIndentingFormatter.test_format_with_timestamp ___________________________________________________________________________

self = <tests.unit.test_logging.TestIndentingFormatter object at 0x7f4ff5514898>, tmpdir = Path('/tmp/pytest-of-ncoghlan/pytest-8/test_format_with_timestamp0')

    def test_format_with_timestamp(self, tmpdir):
        record = logging.makeLogRecord(dict(
            created=1547704837.4,
            msg='hello\nworld',
        ))
        f = IndentingFormatter(fmt="%(message)s", add_timestamp=True)
        expected = '2019-01-17T06:00:37 hello\n2019-01-17T06:00:37 world'
>       assert f.format(record) == expected
E       AssertionError: assert '2019-01-17T1...6:00:37 world' == '2019-01-17T06...6:00:37 world'
E         - 2019-01-17T16:00:37 hello
E         ?            ^
E         + 2019-01-17T06:00:37 hello
E         ?            ^
E         - 2019-01-17T16:00:37 world
E         ?            ^
E         + 2019-01-17T06:00:37 world...
E         
E         ...Full output truncated (2 lines hidden), use '-vv' to show

tests/unit/test_logging.py:68: AssertionError
======================================================================================= short test summary info ========================================================================================
FAIL tests/unit/test_logging.py::TestIndentingFormatter::()::test_format_with_timestamp
================================================================================= 1 failed, 26 passed in 0.15 seconds ==================================================================================
ERROR: InvocationError for command '/home/ncoghlan/devel/pip/.tox/py37/bin/pytest --timeout 300 tests/unit/test_configuration.py tests/unit/test_logging.py' (exited with code 1)
_______________________________________________________________________________________________ summary ________________________________________________________________________________________________
ERROR:   py37: commands failed

The test_base_command errors also look timezone related:

py37 runtests: commands[0] | pytest --timeout 300 tests/unit/test_configuration.py tests/unit/test_base_command.py --show-capture=no
========================================================================================= test session starts ==========================================================================================
platform linux -- Python 3.7.2, pytest-3.8.2, py-1.7.0, pluggy-0.8.1
rootdir: /home/ncoghlan/devel/pip, inifile: setup.cfg
plugins: xdist-1.26.1, timeout-1.3.3, rerunfailures-6.0, forked-1.0.1, cov-2.6.1
timeout: 300.0s
timeout method: signal
timeout func_only: False
collected 28 items                                                                                                                                                                                     

tests/unit/test_configuration.py ......................                                                                                                                                          [ 78%]
tests/unit/test_base_command.py ..FFF.                                                                                                                                                           [100%]

=============================================================================================== FAILURES ===============================================================================================
__________________________________________________________________________ Test_base_command_logging.test_log_command_success __________________________________________________________________________

self = <tests.unit.test_base_command.Test_base_command_logging object at 0x7ff263abeef0>, tmpdir = Path('/tmp/pytest-of-ncoghlan/pytest-8/test_log_command_success0')

    def test_log_command_success(self, tmpdir):
        """
            Test the --log option logs when command succeeds
            """
        cmd = FakeCommand()
        log_path = tmpdir.join('log')
        cmd.main(['fake', '--log', log_path])
        with open(log_path) as f:
>           assert f.read().rstrip() == '2019-01-17T06:00:37 fake'
E           AssertionError: assert '2019-01-17T16:00:37 fake' == '2019-01-17T06:00:37 fake'
E             - 2019-01-17T16:00:37 fake
E             ?            ^
E             + 2019-01-17T06:00:37 fake
E             ?            ^

tests/unit/test_base_command.py:110: AssertionError
___________________________________________________________________________ Test_base_command_logging.test_log_command_error ___________________________________________________________________________

self = <tests.unit.test_base_command.Test_base_command_logging object at 0x7ff263adb048>, tmpdir = Path('/tmp/pytest-of-ncoghlan/pytest-8/test_log_command_error0')

    def test_log_command_error(self, tmpdir):
        """
            Test the --log option logs when command fails
            """
        cmd = FakeCommand(error=True)
        log_path = tmpdir.join('log')
        cmd.main(['fake', '--log', log_path])
        with open(log_path) as f:
>           assert f.read().startswith('2019-01-17T06:00:37 fake')
E           assert False
E            +  where False = <built-in method startswith of str object at 0x56538dad8630>('2019-01-17T06:00:37 fake')
E            +    where <built-in method startswith of str object at 0x56538dad8630> = '2019-01-17T16:00:37 fake\n2019-01-17T16:00:37 Exception:\n2019-01-17T16:00:37 Traceback (most recent call last):\n201...se_command.py", line 16, in run_func\n2019-01-17T16:00:37     raise SystemExit(1)\n2019-01-17T16:00:37 SystemExit: 1\n'.startswith
E            +      where '2019-01-17T16:00:37 fake\n2019-01-17T16:00:37 Exception:\n2019-01-17T16:00:37 Traceback (most recent call last):\n201...se_command.py", line 16, in run_func\n2019-01-17T16:00:37     raise SystemExit(1)\n2019-01-17T16:00:37 SystemExit: 1\n' = <built-in method read of _io.TextIOWrapper object at 0x7ff263c0e8b8>()
E            +        where <built-in method read of _io.TextIOWrapper object at 0x7ff263c0e8b8> = <_io.TextIOWrapper name=Path('/tmp/pytest-of-ncoghlan/pytest-8/test_log_command_error0/log') mode='r' encoding='UTF-8'>.read

tests/unit/test_base_command.py:120: AssertionError
________________________________________________________________________ Test_base_command_logging.test_log_file_command_error _________________________________________________________________________

self = <tests.unit.test_base_command.Test_base_command_logging object at 0x7ff263a71da0>, tmpdir = Path('/tmp/pytest-of-ncoghlan/pytest-8/test_log_file_command_error0')

    def test_log_file_command_error(self, tmpdir):
        """
            Test the --log-file option logs (when there's an error).
            """
        cmd = FakeCommand(error=True)
        log_file_path = tmpdir.join('log_file')
        cmd.main(['fake', '--log-file', log_file_path])
        with open(log_file_path) as f:
>           assert f.read().startswith('2019-01-17T06:00:37 fake')
E           assert False
E            +  where False = <built-in method startswith of str object at 0x56538dac5380>('2019-01-17T06:00:37 fake')
E            +    where <built-in method startswith of str object at 0x56538dac5380> = '2019-01-17T16:00:37 fake\n2019-01-17T16:00:37 Exception:\n2019-01-17T16:00:37 Traceback (most recent call last):\n201...se_command.py", line 16, in run_func\n2019-01-17T16:00:37     raise SystemExit(1)\n2019-01-17T16:00:37 SystemExit: 1\n'.startswith
E            +      where '2019-01-17T16:00:37 fake\n2019-01-17T16:00:37 Exception:\n2019-01-17T16:00:37 Traceback (most recent call last):\n201...se_command.py", line 16, in run_func\n2019-01-17T16:00:37     raise SystemExit(1)\n2019-01-17T16:00:37 SystemExit: 1\n' = <built-in method read of _io.TextIOWrapper object at 0x7ff263c0e7e0>()
E            +        where <built-in method read of _io.TextIOWrapper object at 0x7ff263c0e7e0> = <_io.TextIOWrapper name=Path('/tmp/pytest-of-ncoghlan/pytest-8/test_log_file_command_error0/log_file') mode='r' encoding='UTF-8'>.read

tests/unit/test_base_command.py:130: AssertionError
======================================================================================= short test summary info ========================================================================================
FAIL tests/unit/test_base_command.py::Test_base_command_logging::()::test_log_command_success
FAIL tests/unit/test_base_command.py::Test_base_command_logging::()::test_log_command_error
FAIL tests/unit/test_base_command.py::Test_base_command_logging::()::test_log_file_command_error
================================================================================= 3 failed, 25 passed in 0.17 seconds ==================================================================================
ERROR: InvocationError for command '/home/ncoghlan/devel/pip/.tox/py37/bin/pytest --timeout 300 tests/unit/test_configuration.py tests/unit/test_base_command.py --show-capture=no' (exited with code 1)
_______________________________________________________________________________________________ summary ________________________________________________________________________________________________
ERROR:   py37: commands failed

[ncoghlan]

@cjerdonek It would work, it would just be painful, since the instance constructor expects to receive half a dozen helper functions that specify how to encode and decode keys and values, as well as how to read and write environment variables, and CPython doesn’t directly expose those underlying functions to the Python layer (that’s how os.environ and os.environb are able to be built from a single type that uses the UTF-16-LE OS APIs on Windows, and 8-bit OS APIs everywhere else).

[cjerdonek]

Excellent work narrowing this down, @ncoghlan.

I investigated, and I believe it’s because the ConfigurationMixin class used by test_base_command.py restores os.environ in its teardown() by assigning a copy to os.environ rather than calling clear() followed by update(): https://github.com/pypa/pip/blob/d95b5f2d82c0f52df95b25b09d8e416a721e008e/tests/lib/configuration_helpers.py#L27-L31

This seems to cause the time.tzset() called in TestIndentingFormatter’s setup() not to have the desired effect (I’m guessing because time.tzset() expects os.environ to be the same underlying mapping throughout the process). (Should time.tzset() be immune to this?)

One other place in pip’s code base seems to do this (options_helper.py).

I can file a PR.