Treating warnings as errors may interrupt teardown

Description

Treating warnings as errors (e.g. with option -W error) may interrupt teardown. When warning is reported (with warnings.warn) during teardown, teardown is stopped. Teardown interruption is serious problem which leads to leak of resources (e.g. processes are not closed or temporary files are not deleted).

Expected behavior is teardown completion and then displaying warning message as error.

Minimal example

import warnings

import pytest


@pytest.fixture
def example_fixture():
    yield
    print('Teardown started')
    warnings.warn('Some warning')
    print('Teardown finished')


def test_teardown_bug(example_fixture):
    pass

Without -W error, teardown is fully executed correctly. (Note “Teardown finished” in output below).

$ pytest -s
======================================== test session starts ========================================
platform linux -- Python 3.8.12, pytest-7.0.0, pluggy-1.0.0
rootdir: /home/dev/Desktop/pytest-bug-reproduction
collected 1 item                                                                                    

test_teardown_bug.py .Teardown started
Teardown finished


========================================= warnings summary ==========================================
test_teardown_bug.py::test_teardown_bug
  /home/dev/Desktop/pytest-bug-reproduction/test_teardown_bug.py:10: UserWarning: Some warning
    warnings.warn('Some warning')

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=================================== 1 passed, 1 warning in 0.00s ====================================

When warnings are treated as errors, then teardown is interrupted. (Note that now “Teardown finished” is missing in output below).

$ pytest -s -W error
======================================== test session starts ========================================
platform linux -- Python 3.8.12, pytest-7.0.0, pluggy-1.0.0
rootdir: /home/dev/Desktop/pytest-bug-reproduction
collected 1 item                                                                                    

test_teardown_bug.py .Teardown started
E

============================================== ERRORS ===============================================
______________________________ ERROR at teardown of test_teardown_bug _______________________________

    @pytest.fixture
    def example_fixture():
        yield
        print('Teardown started')
>       warnings.warn('Some warning')
E       UserWarning: Some warning

test_teardown_bug.py:10: UserWarning
====================================== short test summary info ======================================
ERROR test_teardown_bug.py::test_teardown_bug - UserWarning: Some warning
==================================== 1 passed, 1 error in 0.01s =====================================

Environment

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.6 LTS
Release:	18.04
Codename:	bionic

$ python --version
Python 3.8.12

$ pytest --version
pytest 7.0.0

$ pip list
Package    Version
---------- -------
attrs      21.4.0
iniconfig  1.1.1
packaging  21.3
pip        22.0.3
pluggy     1.0.0
py         1.11.0
pyparsing  3.0.7
pytest     7.0.0
setuptools 56.0.0
tomli      2.0.1

[PiotrBatko-BlockchainWares]

I found in warnings documentation that warnings.warn may raise exception (source). I didn’t know about it during writing code using warnings.warn, so my code is non-exception-safe as @asottile wrote. Pytest behavior is now completely understandable, but… 😃

I used -W error, because my intention was to never miss emitted warnings. By default pytest reports at the end of tests, that tests are finished successfully with some warnings and returns 0. In real use cases – on CI or in PyCharm – such warnings are very hard to notice.

I know, that pytest records warnings, because prints information about them in tests summary. For described use case would be enough, if pytest will be able to return non-zero and report about errors instead of warnings. Then CI will be red and IDEs will be able to show that tests failed. It is completely different behavior than -W error, so should be handled by another flag, let’s say --treat-warnings-as-errors, and should be recommended instead of -W error in documentation, because this behavior is what users expects. Complete code execution followed by report about warnings (as errors) instead of interruption in the middle. If user wants interruption, writes raise instead of warn.

[asottile]

ok I don’t think we’re going to agree on that then. anyway my 2c is that pytest should not deviate from how python works here lest we have to explain a complex and difficult-to-understand (and surprisingly different) warning system that deviates from python. I think our recommendation should be that you be careful to write context-safe cleanup (use with / finally) when exceptions could be raised (which isn’t really different from the status quo imo) – and especially when converting warnings to errors