Test functions that return non-None should raise a warning/error

Consider this test:

# The function we're testing
def foo(a: int, b: int) -> int:
  return a * 3 + b

@pytest.mark.parametrize(['a', 'b', 'result'], [
  [1, 2, 5],
  [2, 3, 8],
  [5, 3, 18],
])
def test_foo(a, b, result):
  return foo(a, b) == result

Did you spot the error? The second parametrization has a typo, 2 * 3 + 3 is 9, not 8. But this test actually doesn’t test anything, because it returns the assertion rather than asserting it. This is a common enough mistake, and it wouldn’t normally be a problem except that it can silently cause false positives in test suites.

I propose that test functions that return anything except None fail with a message that cues users that they probably meant to assert rather than return. This feature could be disabled (or enabled, if there are backwards-compatibility issues) via a config flag if necessary.

[asottile]

interestingly enough, unittest in cpython now warns in this situation: https://github.com/python/cpython/pull/27748

[RonnyPfannschmidt]

my note is about refactoring to common functions instead of running other test functions