Amazon AMI Linux returns incorrect value for sysconfig.get_path('purelib') in venvs

Describe the bug

Pipx fails to install any package complaining that pipx_shared.pth file is not found. I looked throughout issues and couldn’t find anything similar.

How to reproduce

1. Create a Cloud9 environment (web IDE)
2. Install pipx: python3 -m pip install pipx --user
3. Install any package

OS, Python, Pipx version:

Admin:~/environment $ python3 --version
Python 3.6.8
Admin:~/environment $ pipx --version
0.14.0.0
Admin:~/environment $ cat /etc/os-release 
NAME="Amazon Linux AMI"
VERSION="2018.03"
ID="amzn"
ID_LIKE="rhel fedora"
VERSION_ID="2018.03"
PRETTY_NAME="Amazon Linux AMI 2018.03"
ANSI_COLOR="0;33"
CPE_NAME="cpe:/o:amazon:linux:2018.03:ga"
HOME_URL="http://aws.amazon.com/amazon-linux-ami/"

Trace with pipx --verbose:

pipx install cfn-lint --verbose
pipx > (run_pipx_command:134): Virtual Environment location is /home/ec2-user/.local/pipx/venvs/cfn-lint
pipx > (run:97): running /usr/bin/python3 -m venv --without-pip /home/ec2-user/.local/pipx/venvs/cfn-lint

pipx > (rmdir:16): removing directory /home/ec2-user/.local/pipx/venvs/cfn-lint
Traceback (most recent call last):
  File "/home/ec2-user/.local/bin/pipx", line 10, in <module>
    sys.exit(cli())
  File "/home/ec2-user/.local/lib/python3.6/site-packages/pipx/main.py", line 547, in cli
    exit(run_pipx_command(parsed_pipx_args))
  File "/home/ec2-user/.local/lib/python3.6/site-packages/pipx/main.py", line 166, in run_pipx_command
    include_dependencies=args.include_deps,
  File "/home/ec2-user/.local/lib/python3.6/site-packages/pipx/commands.py", line 315, in install
    venv.create_venv(venv_args, pip_args)
  File "/home/ec2-user/.local/lib/python3.6/site-packages/pipx/Venv.py", line 128, in create_venv
    pipx_pth.write_text(str(shared_libs.site_packages) + "\n", encoding="utf-8")
  File "/usr/lib64/python3.6/pathlib.py", line 1215, in write_text
    with self.open(mode='w', encoding=encoding, errors=errors) as f:
  File "/usr/lib64/python3.6/pathlib.py", line 1183, in open
    opener=self._opener)
  File "/usr/lib64/python3.6/pathlib.py", line 1037, in _opener
    return self._accessor.open(self, flags, mode)
  File "/usr/lib64/python3.6/pathlib.py", line 387, in wrapped
    return strfunc(str(pathobj), *args)
FileNotFoundError: [Errno 2] No such file or directory: '/home/ec2-user/.local/pipx/venvs/cfn-lint/local/lib/python3.6/site-packages/pipx_shared.pth'

Expected behavior

Install a package

[sahensley]

This is an issue with how Python is configured in the the Amazon Linux AMI environment. The configuration appears to break the venv module in general, not just within pipx. For some reason I have yet to determine, “/local” in injected after the venv name.

The environment also has an undocumented variable exported PYTHON_INSTALL_LAYOUT=amzn which appears to cause other issues. Unsetting this variable and creating a new venv had no impact on the returned path.

Python 3 “purelib” outside of a venv:

ec2-user:~ $ /usr/bin/python3 -c "import sysconfig; print(sysconfig.get_path('purelib'))"
/usr/local/lib/python3.6/site-packages

Python 3 “purelib” inside of a venv:

ec2-user:~ $ python3 -m venv pathtesting
ec2-user:~ $ /home/ec2-user/pathtesting/bin/python -c "import sysconfig; print(sysconfig.get_path('purelib'))"
/home/ec2-user/pathtesting/local/lib/python3.6/site-packages
ec2-user:~ $ /home/ec2-user/pathtesting/bin/python -c "import sysconfig; print(sysconfig.get_paths())"
{'stdlib': '/usr/lib64/python3.6', 'platstdlib': '/home/ec2-user/pathtesting/lib64/python3.6', 'purelib': '/home/ec2-user/pathtesting/local/lib/python3.6/site-packages', 'platlib': '/home/ec2-user/pathtesting/local/lib64/python3.6/site-packages', 'include': '/usr/local/include/python3.6m', 'platinclude': '/usr/local/include/python3.6m', 'scripts': '/home/ec2-user/pathtesting/local/bin', 'data': '/home/ec2-user/pathtesting'}

Python3 “purelib” inside of an activated venv, things work as expected:

ec2-user:~ $ source pathtesting/bin/activate
(pathtesting) ec2-user:~ $ which python3
~/pathtesting/bin/python3
(pathtesting) ec2-user:~ $ python3 -c "import sysconfig; print(sysconfig.get_path('purelib'))"                                                                                                                       
/home/ec2-user/pathtesting/lib/python3.6/site-packages

If you activate the pipx venv via source ~/.local/pipx/shared/bin/activate before you install packages, things work as expected.

ec2-user:~ $ cowsay "The packages continue to work once the pipx venv is deactivated"
  ________________________________________________
 /                                                \
| The packages continue to work once the pipx venv |
| is deactivated                                   |
 \                                                /
  ================================================
                                                     \
                                                      \
                                                        ^__^                             
                                                        (oo)\_______                   
                                                        (__)\       )\/\             
                                                            ||----w |           
                                                            ||     ||  

@cs01 Do you want to keep this open to further workaround the issue? It appears very specific to Amazon’s AMI Linux configuration.

[cs01]

@sahensley thanks for the investigative work. I think the best path forward would be to file an issue somewhere with Amazon about this and to close this as a “won’t fix” since activating a venv before installing is not something pipx can do.