Compact way to combine tests that raise with tests that return values

What’s the problem this feature will solve?

Improved useability in simple cases

Describe the solution you’d like

See code sample and attached examples

Additional context

This is some code that I have written again and again and it removes a lot of boilerplate in writing tests. Seems like it would be a useful addition at source in pytest.

def assert_equal(value, expect, **kwargs):
    """
    Test utility for assorted types, see pd.testing.assert_frame_equal and siblings for supported keywords.

    Parameters
    ----------
    value: the value
    expect: the expectation
    kwargs: passed through depending on the type.

    Raises
    ------
    AssertionError: if not equal
    """
    if isinstance(value, pd.DataFrame):
        assert_frame_equal(value, expect, **kwargs)
    elif isinstance(value, pd.Series):
        assert_series_equal(value, expect, **kwargs)
    elif isinstance(value, pd.Index):
        assert_index_equal(value, expect, **kwargs)
    else:
        assert value == expect

def _is_exception_type(expect):
    return isinstance(expect, type) and issubclass(expect, Exception)

def _is_exception_instance(expect):
    return isinstance(expect, Exception)

def assert_or_raise(func, expect, *args, **kwargs):
    """
    Calls func(*args, **kwargs) and asserts that you get the expected error. If no error is specified,
    returns the value of func(*args, **kwargs)
    """
    if _is_exception_type(expect):
        with pytest.raises(expect):
            func(*args, **kwargs)
    elif _is_exception_instance(expect):
        with pytest.raises(type(expect), match=str(expect)):
            func(*args, **kwargs)
    else:
        return func(*args, **kwargs)


def assert_call(func, expect, *args, test_kwargs: Optional[Dict] = None, **kwargs):
    val = assert_or_raise(func, expect, *args, **kwargs)
    if not (
            _is_exception_type(expect) or _is_exception_instance(expect)
    ):
        assert_equal(val, expect, **(test_kwargs or {}))
    return val

And then you can use it like this:

@pytest.mark.parametrize(
    "input_string, expect",
    [
        (
            "21:00:05.5 [America/New_York]",
            TimeOfDay(
                datetime.time(21, 0, 5, microsecond=500000),
                tz=pytz.timezone("America/New_York")
            )
        ),
        (
            "21,00:05.5 [America/New_York]",
            ValueError(
                "Time string.*did not match.*"
            )
        )
    ]
)
def test_time_of_day_builder(input_string, expect):
    assert_call(
        TimeOfDay.from_str,
        expect,
        input_string
    )

This code now covers a wide variety of easy cases and leads to extremely compact test cases. To summarise the behaviour of assert call is as follows:

• If expect is an exception instance, calls it “with raises” and uses the instance error message as the argument to match.
• If expect is an exception type, calls it with raises but with no argument to match
• If expect is anything else, compares equality. Moreover the equality operator will check for pandas and numpy types, and delegates to their standard equality methods depending on type, allowing you to pass “test_kwargs” such as atol or rtol to the numeric data types.

Without writing functions like this, you end up either having to separate tests that cause errors from tests that do not, or you must put branching logic separately in each test. In my experience the construction above allows you to write much simpler cleaner tests where 90+% of your library unit tests will be a call to assert_call.

[asottile]

#4682 addressed this – though I really think you said the correct thing out loud and that your tests are trying to be too clever:

Without writing functions like this, you end up either having to separate tests that cause errors from tests that do not

this is a good thing

[RonnyPfannschmidt]

@phil20686 i can understand the sentinent for compactness, but its important to keep some domains separate for better understanding

the functions you propose cover a wide range of domains that an individual test writer may not even care about, so all that convenience and deduplication can look like a dreadfull miss-optimization

i intent to get together with @asottile in an effort to provide better data matchers and that hopefully will help to trim down those needs while also helping people to come up with better input/expectation tables

that being said, its still valuable to be able to compactly express the intents when it comes to data expectations however the structure and the implied fuzzyness vs strictness of those tools is important, i have had my fair share of test helpers, that in trying to bee too helpful and too easy, would actually hide test errors/testing issues

my own subjective impression is, that the example helper you showed makes it very easy to err into the direction of deceptive convenience, and that tends to bite very painful