caplog fixture should not be affected by global log level

#7159 made me realize something: I think caplog by default should not be affected by the global log level.

For example:

import logging

def test(caplog):
    logging.info("info")
    logging.critical("critical")
    print(caplog.messages)
    assert 0

Running:

 λ pytest .tmp\test-caplevel.py
======================== test session starts ========================
collected 1 item

.tmp\test-caplevel.py F                                        [100%]

============================= FAILURES ==============================
_______________________________ test ________________________________

caplog = <_pytest.logging.LogCaptureFixture object at 0x000001827FB75358>

    def test(caplog):
        logging.info("info")
        logging.critical("critical")
        logging.debug("debug")
        print(caplog.messages)
>       assert 0
E       assert 0

.tmp\test-caplevel.py:8: AssertionError
----------------------- Captured stdout call ------------------------
['critical']
------------------------- Captured log call -------------------------
CRITICAL root:test-caplevel.py:5 critical
========================= 1 failed in 0.25s =========================

λ pytest .tmp\test-caplevel.py -o log_level=INFO
======================== test session starts ========================
platform win32 -- Python 3.6.6, pytest-5.4.1.dev154+gbe6849644.d20200501, py-1.8.1, pluggy-0.13.0
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('d:\\projects\\pytest\\.hypothesis\\examples')
rootdir: d:\projects\pytest\.tmp, inifile: pytest.ini
plugins: hypothesis-4.36.0, forked-1.1.1, xdist-1.31.0
collected 1 item

.tmp\test-caplevel.py F                                        [100%]

============================= FAILURES ==============================
_______________________________ test ________________________________

caplog = <_pytest.logging.LogCaptureFixture object at 0x0000014A1CCD32E8>

    def test(caplog):
        logging.info("info")
        logging.critical("critical")
        logging.debug("debug")
        print(caplog.messages)
>       assert 0
E       assert 0

.tmp\test-caplevel.py:8: AssertionError
----------------------- Captured stdout call ------------------------
['info', 'critical']
------------------------- Captured log call -------------------------
INFO     root:test-caplevel.py:4 info
CRITICAL root:test-caplevel.py:5 critical
====================== short test summary info ======================
FAILED .tmp\test-caplevel.py::test - assert 0
========================= 1 failed in 0.24s =========================

So depending of the loglevel setting, the test might fail. #7159 is a step in the right direction, because calling caplog.set_level will overwrite the global log level.

But I think this is kind of error prone too, and caplog should have a default log-level value (say INFO), independent from the global log level, which is changed only by calling set_level explicitly.

cc @thisch @ruaridhw

[bluetech]

The way it is currently implemented, caplog doesn’t do anything on its own; it reuses the log capturing that is set up for test reporting.

The semantics I propose are:

• caplog is completely independent from log_level.
• caplog only captures while it (the fixture) is active (independent from runtest hooks).
• caplog captures all levels.

Regarding the last point, @nicoddemus said that the default level should be WARNING, but I think it is more expected for it to capture everything, and the user can assert the level and ignore messages they don’t want to assert. caplog is used specifically to test log messages, I don’t think that if the user wants to test a DEBUG log message, we should require an extra set_level step.

@ruaridhw PR #7159 starts doing this separation but if ⬆️ is what we want, it will require some changes.

@ruaridhw @nicoddemus @thisch WDYT?

[nicoddemus]

I would view this as a fault of the test. I would expect that if the test asserts on a logging message it needs to set caplog.log_level explicitly within the test code. That’s just my opinion though! So in your example, if you require caplog to capture below WARNING, it should explicitly state this.

I understand the reasoning, but I think we should have reasonable defaults to avoid having users writing wrong tests by accident; there’s nothing preventing a user to write a test without setting caplog.log_level and having the test pass, only to break once someone decides to pass --log-level on the command-line (to see different level of captured logs) and having caplog tests fail because of that.

With caplog.log_level having a default value independent from the global value, the average user will be protected in the common case.

Calling pytest on the above code will pass only because of the ini file. Without it, the test will fail because the default is to ignore DEBUG. I believe if we implement this issue, it will be a breaking change because we’re saying the proposed caplog default could be different to the global log level.

Oh you are right, this is a breaking change (forgot to make it explicit), but I believe it is for the best though. It certainly would need to be released in pytest 6.0.0.

Taking this to the extreme, a runner could exec pytest --log_level=100 and every caplog test would fail presuming their tests don’t control caplog’s level themselves

Yes that’s what my proposal tries to avoid. 👍