The virtualenv.pyz zipapp cannot be run with an arbitrary python.

Issue

The virtualenv.pyz zipapp cannot be run with an arbitrary python if that python already has an incompatible version of virtualenv installed. This makes scripting virtualenv hard.

Environment

• OS: Linux
• pip list of the host python where virtualenv is installed: N/A - see Reproduction steps below.

Reproduction Steps

• All good baseline:

$ docker run --rm -it ubuntu:20.04
root@789c91e7e2d3:/# apt update &>/dev/null && apt install -y curl python3 python3-distutils &>/dev/null
root@789c91e7e2d3:/# curl -sSL -O https://raw.githubusercontent.com/pypa/get-virtualenv/20.4.7/public/virtualenv.pyz
root@789c91e7e2d3:/# python3 virtualenv.pyz a.venv
created virtual environment CPython3.8.5.final.0-64 in 274ms
  creator CPython3Posix(dest=/a.venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==21.1.2, setuptools==57.0.0, wheel==0.36.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

• Blow things up by installing virtualenv:

root@789c91e7e2d3:/# apt install -y python3-virtualenv &>/dev/null
root@789c91e7e2d3:/# python3 virtualenv.pyz b.venv
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "virtualenv.pyz/__main__.py", line 168, in <module>
  File "virtualenv.pyz/__main__.py", line 164, in run
  File "virtualenv.pyz/virtualenv/__main__.py", line 18, in run
  File "virtualenv.pyz/virtualenv/run/__init__.py", line 30, in cli_run
  File "virtualenv.pyz/virtualenv/run/__init__.py", line 48, in session_via_cli
  File "virtualenv.pyz/virtualenv/run/__init__.py", line 75, in build_parser
  File "virtualenv.pyz/virtualenv/run/plugin/seeders.py", line 8, in __init__
  File "virtualenv.pyz/virtualenv/run/plugin/base.py", line 39, in options
  File "virtualenv.pyz/virtualenv/run/plugin/base.py", line 18, in entry_points_for
  File "virtualenv.pyz/virtualenv/run/plugin/base.py", line 18, in <genexpr>
  File "/usr/lib/python3.8/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'virtualenv.seed.via_app_data'

This is the same end problem as #1873 but its a bit less under user control. They’ve downloaded the zipapp, run python virtualenv.pyz and got bitten by the fact that python already had virtualenv installed.

Over in Pants where we used to use the virtualenv.pyz to bootstrap a virtualenv for Pants we had to switch to using Pex to run virtualenv since Pex can create a zipapp with proper isolation from the underlying interpreter used to run the PEX zipapp: https://github.com/pantsbuild/setup/pull/99

[mayeut]

A simple workaround that seems to work is to invoke python with the -S flag (“don’t imply ‘import site’ on initialization”)

[jsirois]

Ah, yes Windows (and Python 3.4). I’ll take a crack at a PR to fixup the existing zipapp main module as you suggest.

I do want to note that Pex does support all the rest though. For example (I use --include-tools and PEX_TOOLS=1 here just to have a concise way to demonstrate that multiple versions of distributions are contained in the PEX file and the right ones are activated for the respective interpreter):

$ pex --interpreter-constraint ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*" virtualenv -c virtualenv -o virtualenv.pex --include-tools
$ PEX_TOOLS=1 ./virtualenv.pex info -i4
{
    "always_write_cache": false, 
    "build_properties": {
        "class": "PyPy", 
        "pex_version": "2.1.42", 
        "platform": "manylinux_2_33_x86_64", 
        "version": [
            2, 
            7, 
            18
        ]
    }, 
    "code_hash": "da39a3ee5e6b4b0d3255bfef95601890afd80709", 
    "distributions": {
        "appdirs-1.4.4-py2.py3-none-any.whl": "8b8a15c0d198ccfae8ee432324665d24aeef25e3", 
        "configparser-4.0.2-py2.py3-none-any.whl": "6fdfab75419831b52e6da0d0b0f50fd861cce6fe", 
        "contextlib2-0.6.0.post1-py2.py3-none-any.whl": "7798441a9367996aae652baa225494ccd01f74d2", 
        "distlib-0.3.2-py2.py3-none-any.whl": "3f30172a0b5bce395d4b7dc77cbbd3a219745649", 
        "filelock-3.0.12-py2-none-any.whl": "96e0dee3b9370e1c2648118637217a0c76e1336c", 
        "filelock-3.0.12-py3-none-any.whl": "03d3bd874fa4af18d755d7f2717e3cc9f2f53814", 
        "importlib_metadata-2.1.1-py2.py3-none-any.whl": "eaf8882c291b6c0e51569ae0f7c8c4d0d52cca26", 
        "importlib_metadata-4.5.0-py3-none-any.whl": "301eb5682391c68c48cdbcdf3b04402aadbc5ad0", 
        "importlib_resources-3.2.1-py2.py3-none-any.whl": "d0f724718a374055b6fa092ec475edf53d2381cb", 
        "importlib_resources-3.3.1-py2.py3-none-any.whl": "961a0d57dc2693464538ba13c36c2dffe38e2ca6", 
        "importlib_resources-5.1.4-py3-none-any.whl": "85e09503e1b7d112114c2388063440288e102450", 
        "pathlib2-2.3.5-py2.py3-none-any.whl": "9bfaf31a2df7f99d43d97976de1c2c4414fb9639", 
        "scandir-1.10.0-cp27-cp27mu-linux_x86_64.whl": "e716d31f3544810846e6d32240c22488a0e87467", 
        "scandir-1.10.0-pp27-pypy_73-linux_x86_64.whl": "674c98d613503fc00429bdc583d313224eb8786e", 
        "singledispatch-3.6.2-py2.py3-none-any.whl": "9f516b43610a409bb73b0b756a002d7fd8fafeca", 
        "six-1.16.0-py2.py3-none-any.whl": "035d7c208925c1832def39b592f3477ca36397bf", 
        "typing-3.10.0.0-py2-none-any.whl": "316e046b1c00ceb00c35385c0aca6a874347d671", 
        "typing_extensions-3.10.0.0-py3-none-any.whl": "2d21ea9841868bec222ae7bffdf3212ac3ad9c3e", 
        "virtualenv-20.4.7-py2.py3-none-any.whl": "6a8ad75c3ffc00512cfea6662e102c25fa216cc7", 
        "zipp-1.2.0-py2.py3-none-any.whl": "a90a22bea529301bc39f1de526612414e3cd5dff", 
        "zipp-3.4.1-py3-none-any.whl": "6808f40f165a419dc498ee4e88d5de6d2b9b8ef7"
    }, 
    "emit_warnings": true, 
    "entry_point": "virtualenv.__main__:run_with_catch", 
    "ignore_errors": false, 
    "inherit_path": "false", 
    "interpreter_constraints": [
        ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
    ], 
    "pex_hash": "5961d9d9b4c0146d23c3a7164abb3b84a39d449d", 
    "pex_path": null, 
    "requirements": [
        "virtualenv"
    ], 
    "strip_pex_env": true, 
    "unzip": false, 
    "venv": false, 
    "venv_bin_path": "False", 
    "venv_copies": false, 
    "zip_safe": true, 
    "pex_root": "/home/jsirois/.pex"
}

Under CPython 2.7:

$ PEX_TOOLS=1 python2.7 virtualenv.pex repository info | cut -d' ' -f-2
virtualenv 20.4.7
appdirs 1.4.4
distlib 0.3.2
filelock 3.0.12
six 1.16.0
pathlib2 2.3.5
scandir 1.10.0
importlib-resources 3.3.1
contextlib2 0.6.0.post1
singledispatch 3.6.2
typing 3.10.0.0
zipp 1.2.0
importlib-metadata 2.1.1
configparser 4.0.2

Under PyPy (2.7):

$ PEX_TOOLS=1 pypy virtualenv.pex repository info | cut -d' ' -f-2
virtualenv 20.4.7
appdirs 1.4.4
distlib 0.3.2
filelock 3.0.12
six 1.16.0
pathlib2 2.3.5
scandir 1.10.0
importlib-resources 3.3.1
contextlib2 0.6.0.post1
singledispatch 3.6.2
typing 3.10.0.0
zipp 1.2.0
importlib-metadata 2.1.1
configparser 4.0.2

Note that PyPy gets its own platform-specific wheel activated for scandir.

For CPython 3.5:

$ PEX_TOOLS=1 python3.5 virtualenv.pex repository info | cut -d' ' -f-2
virtualenv 20.4.7
appdirs 1.4.4
distlib 0.3.2
filelock 3.0.12
six 1.16.0
importlib-resources 3.2.1
zipp 1.2.0
importlib-metadata 2.1.1

And for CPython 3.6:

$ PEX_TOOLS=1 python3.6 virtualenv.pex repository info | cut -d' ' -f-2
virtualenv 20.4.7
appdirs 1.4.4
distlib 0.3.2
filelock 3.0.12
six 1.16.0
importlib-resources 5.1.4
zipp 3.4.1
importlib-metadata 4.5.0
typing-extensions 3.10.0.0

Note that different versions of importlib-metadata and importlib-resources are embedded and chosen. Etc.