Installing packages fails if Python 3 installed into path with non-ASCII characters

• Pip version: 9.0.1 (installed with Python 3.6.4)
• Python version: 3.6.4
• Operating system: Windows 7 and 10

Description:

Organizing training where using Python 3 for the first time. Two participants for failed to use pip for anything due to UnicodeDecodeError. After a little debugging it turned out this was due to to them having non-ASCII characters in their user name (not uncommon here in Finland) and Python 3.6 being installed under their account into a path like C:\Users\Käyttäjä\AppData\.... This was the default location offered by the Python installer.

Both users were using Windows 10, but I was able to reproduce this with my Window 7 virtual machine as well. Account with non-ASCII characters isn’t needed, it’s enough to install Python into any path with non-ASCII characters. A workaround was uninstalling Python and installing it directly under C:\.

What I’ve run:

C:\Users\peke>py -m pip --version
pip 9.0.1 from C:\Python36\lib\site-packages (python 3.6)
C:\Users\peke>py -m pip install robotframework
Collecting robotframework
  Using cached robotframework-3.0.2.tar.gz
Installing collected packages: robotframework
  Running setup.py install for robotframework ... error
Exception:
Traceback (most recent call last):
  File "C:\Users\peke\äää\Python36\lib\site-packages\pip\compat\__init__.py", li
ne 73, in console_to_str
    return s.decode(sys.__stdout__.encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe4 in position 23: invalid continuation byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\peke\äää\Python36\lib\site-packages\pip\basecommand.py", line 215, in main
    status = self.run(options, args)
  File "C:\Users\peke\äää\Python36\lib\site-packages\pip\commands\install.py", line 342, in run
    prefix=options.prefix_path,
  File "C:\Users\peke\äää\Python36\lib\site-packages\pip\req\req_set.py", line 784, in install
    **kwargs
  File "C:\Users\peke\äää\Python36\lib\site-packages\pip\req\req_install.py", line 878, in install
    spinner=spinner,
  File "C:\Users\peke\äää\Python36\lib\site-packages\pip\utils\__init__.py", line 676, in call_subprocess
    line = console_to_str(proc.stdout.readline())
  File "C:\Users\peke\äää\Python36\lib\site-packages\pip\compat\__init__.py", line 75, in console_to_str
    return s.decode('utf_8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe4 in position 23: invalid continuation byte

[pradyunsg]

The fix has been released. 😃

The code needs a test though, which is what this issue is tracking.

[pfmoore]

Sorry - I hadn’t hunted down the PR that fixed this. It’s #4486.

Ideally, I’d like our Appveyor tests to run with a non-ASCII username, or possibly a non-latin-1 encoding, but I’ve no idea how we’d do that.