assertion diffs for multiline-string can become unreadable soup

The diff on multiline strings when a string from capysys is differing from expected just by indentation level and/or some leading/trailing whitespace is not great, I think we can do better.

Here’s a reproducer, I just dumped some output of fortune | cowsay into a raw string for example purposes. Intentionally wrong “expected” to demonstrate the issue: it adds 4 space indentation, i.e. hardcoded in a multiline string within a function body but forgetting to use textwrap.dedent on it.

actual = r"""
 ________________________________________
/ You have Egyptian flu: you're going to \
\ be a mummy.                            /
 ----------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||
"""


def test_cowsay_output(capsys):
    print(actual)
    expected = r"""
     ________________________________________
    / You have Egyptian flu: you're going to \
    \ be a mummy.                            /
     ----------------------------------------
            \   ^__^
             \  (oo)\_______
                (__)\       )\/\
                    ||----w |
                    ||     ||
    """
    captured = capsys.readouterr()
    assert captured.out == expected

When running with increased verbosity -vv, it’s often complicated to figure out what the issue is without stepping into a debugger. The initial assertion is on one line which may get wrapped in the terminal over multiple lines:

E       assert ('\n'\n ' ________________________________________\n'\n "/ You have Egyptian flu: you're going to \\\n"\n '\\ be a mummy.                            /\n'\n ' ----------------------------------------\n'\n '        \\   ^__^\n'\n '         \\  (oo)\\_______\n'\n '            (__)\\       )\\/\\\n'\n '                ||----w |\n'\n '                ||     ||\n'\n '\n') == ('\n'\n '     ________________________________________\n'\n "    / You have Egyptian flu: you're going to \\\n"\n '    \\ be a mummy.                            /\n'\n '     ----------------------------------------\n'\n '            \\   ^__^\n'\n '             \\  (oo)\\_______\n'\n '                (__)\\       )\\/\\\n'\n '                    ||----w |\n'\n '                    ||     ||\n'\n '    ')

However the output appears like the actual was a tuple ('\n'\n ' ____ ...

If you look closer you see it is not even valid Python syntax, just a confusing repr corruption, and the multi-line diff that follows isn’t too readable either because of the way the changes are interspersed line by line.

How about an option for the user to suppress pytest’s attempts to make a rich diff, and to just to print one, and then print the other, between some horizontal “margins” such as this?

==================================== ACTUAL ====================================

 ________________________________________
/ You have Egyptian flu: you're going to \
\ be a mummy.                            /
 ----------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

=================================== EXPECTED ===================================

     ________________________________________
    / You have Egyptian flu: you're going to \
    \ be a mummy.                            /
     ----------------------------------------
            \   ^__^
             \  (oo)\_______
                (__)\       )\/\
                    ||----w |
                    ||     ||
    
================================================================================

[wimglenn]

My preference would be for a vertical presentation, not side-by-side. It has a few wins in basic practicality

• ease of copy-pasting the content
• no complications for long lines (side-by-side necessitates double the terminal width avail, or horizontal scrollbar)
• simpler to implement

I also don’t want to see any extra diff characters inline like “E” on the start of the line or stuff like “?”, “-”, “++++” within the presentation.

[blueyed]

One good heuristic here would also be to try with indentation removed (textwrap.dedent), and report it as “matches with different indent”.

Ratio is also good, of course.

FWIW I have this in my conftest to print more with verbosity 3 - very helpful to adjust existing tests:

@pytest.hookimpl(hookwrapper=True)
def pytest_assertrepr_compare(config: Config, op: str, left: Any, right: Any) -> Optional[List[str]]:
    verbose = config.option.verbose

    outcome = yield
    result = outcome.get_result()  # type: Optional[List[str]]
    if not result:
        return
    lines = result[0]

    …

    if verbose > 2:
        import pprint
        full = pprint.saferepr(right).splitlines()
        output = [
            "Expected:",
        ] + full
        lines.extend(output)
        full = pprint.saferepr(left).splitlines()
        output = [
            "Actual:",
        ] + full
        lines.extend(output)
        print(left)

    outcome.force_result(result)

does pytest have any opinion, introspection, or convention about which is the “actual” and which is the “expected”? I suppose it might have to be just be presented “left” and then “right”.

See #3333 / d59adc61f