Asserting repeated warning does not work in Python 2.7

Running the following test code with Python 2.7 will fail because for some reason the warning is not emitted a second time:

import warnings
import pytest

def warn():
    warnings.warn(UserWarning("test"))

def test_warning():
    warn()
    with pytest.warns(UserWarning):
        warn()

The output:

============================= test session starts ==============================
platform linux2 -- Python 2.7.13, pytest-3.2.3, py-1.4.34, pluggy-0.4.0
rootdir: /home/jgosmann, inifile:
collected 1 item

test2.py F

=================================== FAILURES ===================================
_________________________________ test_warning _________________________________

    def test_warning():
        warn()
        with pytest.warns(UserWarning):
>           warn()
E           Failed: DID NOT WARN. No warnings of type (<type 'exceptions.UserWarning'>,) was emitted. The list of emitted warnings is: [].

test2.py:13: Failed
=============================== warnings summary ===============================
test2.py::test_warning
  /home/jgosmann/test2.py:7: UserWarning: test
    warnings.warn(UserWarning("test"))

-- Docs: http://doc.pytest.org/en/latest/warnings.html
===================== 1 failed, 1 warnings in 0.01 seconds =====================

Running with Python 3.6.1 succeeds:

============================= test session starts ==============================
platform linux -- Python 3.6.1, pytest-3.2.3, py-1.4.34, pluggy-0.4.0
rootdir: /home/jgosmann, inifile:
collected 1 item

test2.py .

=============================== warnings summary ===============================
test2.py::test_warning
  /home/jgosmann/test2.py:7: UserWarning: test
    warnings.warn(UserWarning("test"))

-- Docs: http://doc.pytest.org/en/latest/warnings.html
===================== 1 passed, 1 warnings in 0.00 seconds =====================

Note that this also occurs across different tests, e.g. when parametrizing:

@pytest.mark.parametrize('i', range(2))
def test_warning(i):
    if i % 2 == 0:
        warn()
    else:
        with pytest.warns(UserWarning):
            warn()

When running tests with pytest-xdist in non-deterministic order this can cause random test failures with Python 2.7.

[jgosmann]

This problem also occurs for non-deprecation warnings. Maybe pytest.warns should capture warnings in the same way as pytest.deprecated_call?

[asottile]

via #4104