[Windows] Could not install packages due to an EnvironmentError: [Errno 42] Illegal byte sequence

Environment

• pip version: 18.0
• Python version: 2.7.15 64 bit
• OS: Windows 10.0.17134.165 x64
• The console locale is set to cp872.
• The console font is consolas.

Description pip throws an exception when trying to display a progress bar for the download of any package on Windows.

Expected behavior pip should work and install the package correctly.

How to Reproduce

Note that this is environment dependent.

1. pip install -I --no-cache-dir --verbose wheel

Using --progress-bar ascii doesn’t help. --progress-bar off works, but is sub-optimal as you get no progress.

Analysis The following seems to succeed, while later attempts to actually write the characters to the console fail. 😖 https://github.com/pypa/pip/blob/3e81d8cde142687deda8fb5a07ca3fb38906b007/src/pip/_internal/utils/ui.py#L52-L60

It will work on Python 3 since it has unicode support for the console. There is win-unicode-console which could be used to get similar support for Python 2. But the check should probably be fixed anyhow.

Output

>pip install -I --no-cache-dir --verbose wheel
Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
Config variable 'WITH_PYMALLOC' is unset, Python ABI tag may be incorrect
Config variable 'Py_UNICODE_SIZE' is unset, Python ABI tag may be incorrect
Created temporary directory: c:\users\user\appdata\local\temp\pip-ephem-wheel-cache-5axu12
Created temporary directory: c:\users\user\appdata\local\temp\pip-req-tracker-r7iyls
Created requirements tracker 'c:\\users\\user\\appdata\\local\\temp\\pip-req-tracker-r7iyls'
Created temporary directory: c:\users\user\appdata\local\temp\pip-install-5priwk
Collecting wheel
  1 location(s) to search for versions of wheel:
  * https://pypi.org/simple/wheel/
  Getting page https://pypi.org/simple/wheel/
  Starting new HTTPS connection (1): pypi.org:443
  https://pypi.org:443 "GET /simple/wheel/ HTTP/1.1" 200 7077
  Analyzing links from page https://pypi.org/simple/wheel/
    Found link https://files.pythonhosted.org/packages/51/15/d31364732deb6e7d1490f8a08f011997b52b4b106d592d36e917404be613/wheel-0.1.tar.gz#sha256=2cc0dd3d9465ae323ad6e85fc9dc54ab3664da2c60ce42189b9ae4c44de7d8bd (from https://pypi.org/simple/wheel/), version: 0.1
    Found link https://files.pythonhosted.org/packages/e2/1e/aedf6a9c6c0a588f96e9fe6e953de0c9f9084baad05f3ed0bd4a9b4a469f/wheel-0.2.tar.gz#sha256=82026a421ca379affefa9a0cb85807047e7184574a92f406670b2dcc3384da36 (from https://pypi.org/simple/wheel/), version: 0.2
    <snip>
    Found link https://files.pythonhosted.org/packages/81/30/e935244ca6165187ae8be876b6316ae201b71485538ffac1d718843025a9/wheel-0.31.1-py2.py3-none-any.whl#sha256=80044e51ec5bbf6c894ba0bc48d26a8c20a9ba629f4ca19ea26ecfcf87685f5f (from https://pypi.org/simple/wheel/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*), version: 0.31.1
    Found link https://files.pythonhosted.org/packages/2a/fb/aefe5d5dbc3f4fe1e815bcdb05cbaab19744d201bbc9b59cfa06ec7fc789/wheel-0.31.1.tar.gz#sha256=0a2e54558a0628f2145d2fc822137e322412115173e8a2ddbe1c9024338ae83c (from https://pypi.org/simple/wheel/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*), version: 0.31.1
  Using version 0.31.1 (newest of versions: 0.1, 0.2, 0.3, 0.4, 0.4.1, 0.4.2, 0.5, 0.6, 0.7, 0.8, 0.9, 0.9.1, 0.9.2, 0.9.3, 0.9.4, 0.9.5, 0.9.6, 0.9.7, 0.10.0, 0.10.1, 0.10.2, 0.10.3, 0.11.0, 0.12.0, 0.13.0, 0.14.0, 0.15.0, 0.16.0, 0.17.0, 0.18.0, 0.19.0, 0.21.0, 0.22.0, 0.23.0, 0.24.0, 0.25.0, 0.26.0, 0.27.0, 0.28.0, 0.29.0, 0.30.0, 0.31.0, 0.31.1)
  Created temporary directory: c:\users\user\appdata\local\temp\pip-unpack-euk6au
  Starting new HTTPS connection (1): files.pythonhosted.org:443
  https://files.pythonhosted.org:443 "GET /packages/81/30/e935244ca6165187ae8be876b6316ae201b71485538ffac1d718843025a9/wheel-0.31.1-py2.py3-none-any.whl HTTP/1.1" 200 41567
  Downloading https://files.pythonhosted.org/packages/81/30/e935244ca6165187ae8be876b6316ae201b71485538ffac1d718843025a9/wheel-0.31.1-py2.py3-none-any.whl (41kB)
  Downloading from URL https://files.pythonhosted.org/packages/81/30/e935244ca6165187ae8be876b6316ae201b71485538ffac1d718843025a9/wheel-0.31.1-py2.py3-none-any.whl#sha256=80044e51ec5bbf6c894ba0bc48d26a8c20a9ba629f4ca19ea26ecfcf87685f5f (from https://pypi.org/simple/wheel/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*)
    24% |Could not install packages due to an EnvironmentError.
Traceback (most recent call last):
  File "c:\python27\lib\site-packages\pip\_internal\commands\install.py", line 299, in run
    resolver.resolve(requirement_set)
  File "c:\python27\lib\site-packages\pip\_internal\resolve.py", line 102, in resolve
    self._resolve_one(requirement_set, req)
  File "c:\python27\lib\site-packages\pip\_internal\resolve.py", line 256, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "c:\python27\lib\site-packages\pip\_internal\resolve.py", line 209, in _get_abstract_dist_for
    self.require_hashes
  File "c:\python27\lib\site-packages\pip\_internal\operations\prepare.py", line 283, in prepare_linked_requirement
    progress_bar=self.progress_bar
  File "c:\python27\lib\site-packages\pip\_internal\download.py", line 836, in unpack_url
    progress_bar=progress_bar
  File "c:\python27\lib\site-packages\pip\_internal\download.py", line 673, in unpack_http_url
    progress_bar)
  File "c:\python27\lib\site-packages\pip\_internal\download.py", line 897, in _download_http_url
    _download_url(resp, link, content_file, hashes, progress_bar)
  File "c:\python27\lib\site-packages\pip\_internal\download.py", line 617, in _download_url
    hashes.check_against_chunks(downloaded_chunks)
  File "c:\python27\lib\site-packages\pip\_internal\utils\hashes.py", line 48, in check_against_chunks
    for chunk in chunks:
  File "c:\python27\lib\site-packages\pip\_internal\download.py", line 585, in written_chunks
    for chunk in chunks:
  File "c:\python27\lib\site-packages\pip\_internal\utils\ui.py", line 161, in iter
    self.next(n)
  File "c:\python27\lib\site-packages\pip\_vendor\progress\__init__.py", line 73, in next
    self.update()
  File "c:\python27\lib\site-packages\pip\_vendor\progress\bar.py", line 86, in update
    self.writeln(line)
  File "c:\python27\lib\site-packages\pip\_vendor\progress\helpers.py", line 68, in writeln
    print(line, end='', file=self.file)
  File "c:\python27\lib\site-packages\pip\_vendor\colorama\ansitowin32.py", line 141, in write
    self.write_and_convert(text)
  File "c:\python27\lib\site-packages\pip\_vendor\colorama\ansitowin32.py", line 169, in write_and_convert
    self.write_plain_text(text, cursor, len(text))
  File "c:\python27\lib\site-packages\pip\_vendor\colorama\ansitowin32.py", line 174, in write_plain_text
    self.wrapped.write(text[start:end])
IOError: [Errno 42] Illegal byte sequence
Cleaning up...
Removed build tracker 'c:\\users\\user\\appdata\\local\\temp\\pip-req-tracker-r7iyls'
There was an error checking the latest version of pip
Traceback (most recent call last):
  File "c:\python27\lib\site-packages\pip\_internal\utils\outdated.py", line 90, in pip_version_check
    state = SelfCheckState(cache_dir=options.cache_dir)
  File "c:\python27\lib\site-packages\pip\_internal\utils\outdated.py", line 25, in __init__
    self.statefile_path = os.path.join(cache_dir, "selfcheck.json")
  File "c:\python27\lib\ntpath.py", line 65, in join
    result_drive, result_path = splitdrive(path)
  File "c:\python27\lib\ntpath.py", line 115, in splitdrive
    if len(p) > 1:
TypeError: object of type 'bool' has no len()

P.S. The second exception is an entirely different bug… This happens when you use --no-cache-dir. Not sure if reported, if not, then a separate issue needs to be opened for that.

[segevfiner]

--progress-bar off also works.

[undera]

One workaround that worked for me: use pip install <package> -q. This made pip to not display progressbar which were triggering issue. I had no output displayed, but at least it has installed the package.