http: error: UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 1617: character maps to <undefined> when re-direction output to disk

On Windows 10, I’m using httpie to retrieve data. It works fine when displaying on the screen, but as soon as I pipe the output to disk, I get http: error: UnicodeDecodeError: ‘charmap’ codec can’t decode byte 0x9d in position 8024: character maps to <undefined>

I tried to attach the file I’m having the problem with, but unfortunately, it 100Mb

Here is the complete debug output:

C:\Users\PeterKronenberg\Desktop>http --debug --print=Bb --pretty=all --session=C:\Users\PETERK~1\AppData\Local\Temp/access.json http://localhost:8080/nexus/extract/path  filePath=c:/TestFiles/100Mb.pdf > foo
HTTPie 2.3.0
Requests 2.25.1
Pygments 2.7.3
Python 3.9.1 (tags/v3.9.1:1e5d33e, Dec  7 2020, 17:08:21) [MSC v.1927 64 bit (AMD64)]
c:\python\python39\python.exe
Windows 10

<Environment {'colors': 256,
 'config': {'default_options': []},
 'config_dir': WindowsPath('C:/Users/PeterKronenberg/AppData/Roaming/httpie'),
 'devnull': <property object at 0x000001112C1815E0>,
 'is_windows': True,
 'log_error': <function Environment.log_error at 0x000001112C1AC1F0>,
 'program_name': 'http',
 'stderr': <colorama.ansitowin32.StreamWrapper object at 0x000001112C176F10>,
 'stderr_isatty': True,
 'stdin': <_io.TextIOWrapper name='<stdin>' mode='r' encoding='utf-8'>,
 'stdin_encoding': 'utf-8',
 'stdin_isatty': True,
 'stdout': <colorama.ansitowin32.StreamWrapper object at 0x000001112C176BE0>,
 'stdout_encoding': 'cp1252',
 'stdout_isatty': False}>

>>> requests.request(**{'auth': None,
 'data': '{"filePath": "c:/TestFiles/100Mb.pdf"}',
 'headers': {'User-Agent': b'HTTPie/2.3.0', 'Accept': b'application/json, */*;q=0.5', 'Content-Type': b'application/json', 'authorization': b'eyJhbGciOiJSUzI1NiIsImtpZCI6InB1YmxpYzplMjM3ODE1Ny04MmY0LTQ2YjItYWI5NS1hMDRhMzg4ZGY5ZTQiLCJ0eXAiOiJKV1QifQ.eyJhdWQiOltdLCJjbGllbnRfaWQiOiIxMTExMTExMS0xMTExLTExMTEtMTExMS0xMTExMTExMTExMTUiLCJleHAiOjE2MDk0NDAxNzYsImV4dCI6e30sImlhdCI6MTYwOTQzNjU3NiwiaXNzIjoiaHR0cHM6Ly9vaWRjLnRvcmNoLmFpLyIsImp0aSI6IjZkN2QwOTFkLTQyMTctNGRmMS04ZmY3LTk2MDdiMzkzZjBlZCIsIm5iZiI6MTYwOTQzNjU3Niwic2NwIjpbXSwic3ViIjoiMTExMTExMTEtMTExMS0xMTExLTExMTEtMTExMTExMTExMTE1In0.QeCNhpoE-1UyLcHer2_JjhiMx3qKV1qIC3LX0rlRUKSIK5vlMbJ2KmVP1mNR647SJhAxN9zdqf9vDaG6Fk0ZRLBFSBbcq0WuyTlaJsPVVggpe9ys_mcqO8uPMTl4-tO0H5kXVzRi_TFcmfCH8W3X8KAe3K7kjfs8O207EAenPr8n-KB12qUQgpC4ELXRbC_9SwuZY_ZhOic9POUYMCvrmJkMB8Th4kOqix5uI45lwowVQI9kTgAB4kQF8VelYJF7xxxn81U24T3ab9lRtp6ShY1mUkhDwfhrz2YdhYQ4uhQhKNZc3Hxz8OPSV_0Q-ykdZVgi03su51E_SE0odnryiUrR4nheu1cxmGmKNq8gb2CII9NktgrZEvvXlYJ5cV5mI2AAVlgkp4A0FBCd_wt39mXadIWPjv3YfilkzwUZCoeKA8dpp1mOKebFjq7Iw4LnxhObuKw1BnhYjHzdtQC5mhwFtc5iCeYChkQG_b8vybap12S4HzQ7HHFBKjdJZ2Il5mbuF01BAUBhZ1TO2GQ_t3O0JVrp8rmob75Yiza5FiBZaeAASSp2p0mb-_CmTVnQq3N-f099yL7byNYKl0toLIfgDLuib6bv_rhushdvEyWOD1c-nJnqMNFekEZ4NXTvBskL9PJCWcW-r8l1Fn567s7SAFan-G37YeQYBfmwADI'},
 'method': 'post',
 'params': <generator object MultiValueOrderedDict.items at 0x000001112CA5FB30>,
 'url': 'http://localhost:8080/nexus/extract/path'})


http: error: UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 1617: character maps to <undefined>

Traceback (most recent call last):
  File "c:\python\python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\python\python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Python\Python39\Scripts\http.exe\__main__.py", line 7, in <module>
  File "c:\python\python39\lib\site-packages\httpie\__main__.py", line 11, in main
    exit_status = main()
  File "c:\python\python39\lib\site-packages\httpie\core.py", line 79, in main
    exit_status = program(
  File "c:\python\python39\lib\site-packages\httpie\core.py", line 222, in program
    write_message(
  File "c:\python\python39\lib\site-packages\httpie\output\writer.py", line 38, in write_message
    write_stream_with_colors_win_py3(**write_stream_kwargs)
  File "c:\python\python39\lib\site-packages\httpie\output\writer.py", line 82, in write_stream_with_colors_win_py3
    outfile.write(chunk.decode(encoding))
  File "c:\python\python39\lib\encodings\cp1252.py", line 15, in decode
    return codecs.charmap_decode(input,errors,decoding_table)
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 1617: character maps to <undefined>

[Almad]

I think it may make sense to do it together / following the similar algorithm as #1022 for mismatches

[jakubroztocil]

Guidelines for a future fix:

1. That issue relates specifically to colors, which can be enabled with both --format=all and --format=colors
2. It is specific to Windows.
3. It may have to do more with how colors are represented on Windows than with an encoding mismatch.

I’d find out what actually gets written to the file when colors are applied. And then see if it even makes sense to allow non-terminal coloured output on Windows: does cat ./output correctly reproduce the coloured output? Perhaps the best solution will be simply disabling colors on Windows unless the output is a terminal.