pytest 6 (?): UnicodeEncodeError in terminalwriter on Windows

Just got this on GitHub Actions with the pytest 6 rc:

__________ test_crash_when_pasting_emoji_into_the_command_line_2007 ___________

request = <FixtureRequest for <Function test_crash_when_pasting_emoji_into_the_command_line_2007>>

    @pytest.mark.usefixtures(*function_args)
    def scenario_wrapper(request):
>       _execute_scenario(feature, scenario, request, encoding)

.tox\py37-pyqt514\lib\site-packages\pytest_bdd\scenario.py:200: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
.tox\py37-pyqt514\lib\site-packages\pytest_bdd\scenario.py:166: in _execute_scenario
    _execute_step_function(request, scenario, step, step_func)
.tox\py37-pyqt514\lib\site-packages\pytest_bdd\scenario.py:115: in _execute_step_function
    step_func(**kwargs)
ests\end2end\features\conftest.py:279: in run_command
    quteproc.send_cmd(command, count=count, invalid=invalid)
ests\end2end\fixtures\quteprocess.py:748: in send_cmd
    self.send_ipc([command])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <end2end.fixtures.quteprocess.QuteProc object at 0x00000218EC2C61F8>
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\_pytest\main.py", line 240, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\_pytest\main.py", line 296, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\pluggy\hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\pluggy\manager.py", line 87, in <lambda>
INTERNALERROR>     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\pluggy\callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\pluggy\callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\pluggy\callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\_pytest\main.py", line 321, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\pluggy\hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\pluggy\manager.py", line 87, in <lambda>
INTERNALERROR>     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\pluggy\callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\pluggy\callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\pluggy\callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\_pytest\runner.py", line 100, in pytest_runtest_protocol
INTERNALERROR>     runtestprotocol(item, nextitem=nextitem)
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\_pytest\runner.py", line 117, in runtestprotocol
INTERNALERROR>     reports.append(call_and_report(item, "call", log))
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\_pytest\runner.py", line 211, in call_and_report
INTERNALERROR>     hook.pytest_runtest_logreport(report=report)
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\pluggy\hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\pluggy\manager.py", line 87, in <lambda>
INTERNALERROR>     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\pluggy\callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\pluggy\callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\pluggy\callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\pytest_instafail.py", line 60, in pytest_runtest_logreport
INTERNALERROR>     self.print_failure(report)
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\pytest_instafail.py", line 89, in print_failure
INTERNALERROR>     self._outrep_summary(report)
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\_pytest\terminal.py", line 1035, in _outrep_summary
INTERNALERROR>     rep.toterminal(self._tw)
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\_pytest\reports.py", line 82, in toterminal
INTERNALERROR>     longrepr.toterminal(out)
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\_pytest\_code\code.py", line 967, in toterminal
INTERNALERROR>     element[0].toterminal(tw)
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\_pytest\_code\code.py", line 997, in toterminal
INTERNALERROR>     entry.toterminal(tw)
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\_pytest\_code\code.py", line 1093, in toterminal
INTERNALERROR>     self.reprfuncargs.toterminal(tw)
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\_pytest\_code\code.py", line 1156, in toterminal
INTERNALERROR>     tw.line(linesofar)
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\_pytest\_io\terminalwriter.py", line 156, in line
INTERNALERROR>     self.write(s, **markup)
INTERNALERROR>   File "D:\a\qutebrowser\qutebrowser\.tox\py37-pyqt514\lib\site-packages\_pytest\_io\terminalwriter.py", line 151, in write
INTERNALERROR>     self._file.write(msg)
INTERNALERROR>   File "c:\hostedtoolcache\windows\python\3.7.8\x64\lib\encodings\cp1252.py", line 19, in encode
INTERNALERROR>     return codecs.charmap_encode(input,self.errors,encoding_table)[0]
INTERNALERROR> UnicodeEncodeError: 'charmap' codec can't encode character '\U0001f300' in position 31: character maps to <undefined>

This is the test using pytest-bdd in qutebrowser:

    Scenario: Crash when pasting emoji into the command line (#2007)
        Given I open about:blank
        When I run :set-cmd-text -s :🌀
        Then no crash should happen

I can’t reproduce this consistently - if I’m reading the traceback right, the test failed (probably flaky?) and then pytest failed while printing the traceback… Note that I also have pytest-instafail installed and it shows up in the stacktrace.

Not sure if this is actually related to pytest 6, but I’ve never seen it happen before.

[bluetech]

OK, I’ll bring back the escaping, especially because I failed to mention this change it in the commit/changelog.

(Note: I changed the issue title UncodeDecodeError -> UnicodeEncodeError).

[nicoddemus]

While I can sympathize with @bluetech’s sentiment of exposing the misconfiguration, unfortunately I think we should get back escape strings here. This example by @The-Compiler nails it down for me:

def func(text):
    assert False

def test_func():
    func('\U0001f300')

The user is not even printing to the console, so it is not helpful that pytest breaks because it is writing to the console, and worse only when the test fails, making the problem possibly go silent and blowing up much later on a different host.

So 👍 from me to escape this again (about how to do it I don’t have a strong opinion).