Docs on warnings filters should note that `ResourceWarning` is often delayed

Some of my tests need to ensure open files are closed. Let’s use the simplest example:

# test_example.py
import pytest

@pytest.mark.filterwarnings('error::ResourceWarning')
def test_resourcewarning():
    open('/dev/null')

When I run pytest, the warning is thrown and even printed in the test output, but the test still passes:

$ pytest
======================================== test session starts =========================================
platform darwin -- Python 3.10.2, pytest-7.1.1, pluggy-1.0.0
rootdir: /path/to/project
collected 1 item

test_example.py .                                                                              [100%]

========================================== warnings summary ==========================================
test_example.py::test_resourcewarning
  /path/to/python/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning: 
Exception ignored in: <_io.FileIO [closed]>

  Traceback (most recent call last):
    File "/path/to/project/test_example.py", line 5, in test_resourcewarning
      open('/dev/null')
  ResourceWarning: unclosed file <_io.TextIOWrapper name='/dev/null' mode='r' encoding='UTF-8'>

    warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
==================================== 1 passed, 1 warning in 0.01s ====================================

Removing @pytest.mark.filterwarnings or changing the warning type to something unrelated (like DeprecationWarning) results in the test passing without printing any warnings at all. That tells me pytest is picking up the warning, but it’s being subsequently caught by pytest.PytestUnraisableExceptionWarning, and my tests still pass because I wasn’t filtering for that. If I filter for pytest.PytestUnraisableExceptionWarning instead the test also passes, because it isn’t looking for the original ResourceWarning.

The only solution I can think of is to filter for both:

@pytest.mark.filterwarnings('error::ResourceWarning')
@pytest.mark.filterwarnings('error::pytest.PytestUnraisableExceptionWarning')

Unless I’m missing something this seems to be a bug for ResourceWarning in particular, since I can’t reproduce this with other warning types. I think failing the test case without requiring the 2nd generic warning filter is the reasonable expected behaviour here.

Note: this test example was run on a vanilla virtual env:

$ pip freeze
attrs==21.4.0
iniconfig==1.1.1
packaging==21.3
pluggy==1.0.0
py==1.11.0
pyparsing==3.0.7
pytest==7.1.1
tomli==2.0.1

[dchevell]

Your explanation of expected behaviour is precisely what I was looking for, without that the change to docs wasn’t very clear.

Agreed on the explanatory section, perhaps the solution of filtering in combination with 'error::pytest.PytestUnraisableExceptionWarning' would fit there as well.

[Zac-HD]

It’s not behaving how you expect, but it is behaving how I expect it to - and IMO that means it’s a docs issue. And yes, unfortunately it’s not feasible to change this, because the ResourceWarning literally doesn’t exist in the scope that we want to handle it; the warning is only raised much later during garbage collection. In fact the language doesn’t guarantee that you’ll ever get a ResourceWarning, and so sometimes (often, on PyPy) you simply won’t!

So I think an explanatory section much like the “DeprecationWarning and PendingDeprecationWarning” section, and located just below it", would be useful here 🙂