Discussion about how subtests failures should be displayed

One more test case that is interesting. This file:

import unittest

class T(unittest.TestCase):
    def test_fail(self):
        with self.subTest():
           self.assertEqual(1, 2)

No passing subtests, just one failure. Still shows PASSED. I would expect FAILED and the bottom line to show 1 failed Maybe pytest itself is calling pytest_runtest_logreport() at the end and causing an extra test to be counted.

pytest -v test_sub.py
=============================== test session starts ===============================
platform win32 -- Python 3.7.1, pytest-4.4.0, ...
plugins: subtests-0.2.0
collected 1 item

test_sub.py::T::test_fail FAILED                                         [100%]
test_sub.py::T::test_fail PASSED                                             [100%]

==================================== FAILURES =====================================
_____________________________ T.test_fail (<subtest>) _____________________________

self = <test_sub.T testMethod=test_fail>

    def test_fail(self):
        with self.subTest():
>          self.assertEqual(1, 2)
E          AssertionError: 1 != 2

test_sub.py:6: AssertionError
======================= 1 failed, 1 passed in 0.05 seconds ========================

_Originally posted by @okken in https://github.com/pytest-dev/pytest-subtests/issues/7#issuecomment-479983466_

cc @jurisbu @bskinn

[nicoddemus]

(I’m short on time so I will read your post more carefully later @bskinn, thanks!)

This relates to a broader pain I’ve been feeling, where AFAIK pytest has no mechanism for selectively marking test functions as xfail (“@pytest.mark.xfail_if(…)”, so to speak).

Just wanted to mention that xfail supports a condition argument for exactly that purpose. 😉

[bskinn]

🤦‍♂️ Thank you. 😃