Virtualenv corrupts `dist-info` during embedded downgrade overlaying existing venv

Scenario:

1. Virtualenv carries outdated embedded setuptools (or any other internal essential).
2. Rollout virtualenv.
3. Activate rolled out venv.
4. Upgrade setuptools with pip.
5. Deactivate venv.
6. Rollout virtualenv again on top of the existing venv.
7. Observe multiple dist-info directories present for incorrectly downgraded tool, resulting in venv corruption.

Example - always reproducible:

(pyb-3.9.7) [username@hostname pybuilder]$ pip install virtualenv\<20.8
Collecting virtualenv<20.8
  Using cached virtualenv-20.7.2-py2.py3-none-any.whl (5.3 MB)
Requirement already satisfied: distlib<1,>=0.3.1 in /home/username/.pyenv/versions/3.9.7/envs/pyb-3.9.7/lib/python3.9/site-packages (from virtualenv<20.8) (0.3.2)
Requirement already satisfied: backports.entry-points-selectable>=1.0.4 in /home/username/.pyenv/versions/3.9.7/envs/pyb-3.9.7/lib/python3.9/site-packages (from virtualenv<20.8) (1.1.0)
Requirement already satisfied: platformdirs<3,>=2 in /home/username/.pyenv/versions/3.9.7/envs/pyb-3.9.7/lib/python3.9/site-packages (from virtualenv<20.8) (2.3.0)
Requirement already satisfied: six<2,>=1.9.0 in /home/username/.pyenv/versions/3.9.7/envs/pyb-3.9.7/lib/python3.9/site-packages (from virtualenv<20.8) (1.16.0)
Requirement already satisfied: filelock<4,>=3.0.0 in /home/username/.pyenv/versions/3.9.7/envs/pyb-3.9.7/lib/python3.9/site-packages (from virtualenv<20.8) (3.0.12)
Installing collected packages: virtualenv
  Attempting uninstall: virtualenv
    Found existing installation: virtualenv 20.8.0
    Uninstalling virtualenv-20.8.0:
      Successfully uninstalled virtualenv-20.8.0
Successfully installed virtualenv-20.7.2
(pyb-3.9.7) [username@hostname pybuilder]$ virtualenv test-env
created virtual environment CPython3.9.7.final.0-64 in 113ms
  creator CPython3Posix(dest=/home/username/src/pybuilder/test-env, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/username/.local/share/virtualenv)
    added seed packages: pip==21.2.3, setuptools==57.4.0, wheel==0.37.0
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator
(pyb-3.9.7) [username@hostname pybuilder]$ source test-env/bin/activate
(test-env) (pyb-3.9.7) [username@hostname pybuilder]$ pip install -U setuptools
Requirement already satisfied: setuptools in ./test-env/lib/python3.9/site-packages (57.4.0)
Collecting setuptools
  Using cached setuptools-58.0.4-py3-none-any.whl (816 kB)
Installing collected packages: setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 57.4.0
    Uninstalling setuptools-57.4.0:
      Successfully uninstalled setuptools-57.4.0
Successfully installed setuptools-58.0.4
WARNING: You are using pip version 21.2.3; however, version 21.2.4 is available.
You should consider upgrading via the '/home/username/src/pybuilder/test-env/bin/python -m pip install --upgrade pip' command.
(test-env) (pyb-3.9.7) [username@hostname pybuilder]$ deactivate
(pyb-3.9.7) [username@hostname pybuilder]$ ls -la test-env/lib/python3.9/site-packages/
total 24
drwxr-xr-x 11 username username 4096 Sep 16 13:34 .
drwxr-xr-x  3 username username   27 Sep 16 13:34 ..
drwxr-xr-x  3 username username   63 Sep 16 13:34 _distutils_hack
-rw-r--r--  1 username username  152 Sep 16 13:34 distutils-precedence.pth
drwxr-xr-x  5 username username  111 Sep 16 13:34 pip
drwxr-xr-x  2 username username  130 Sep 16 13:34 pip-21.2.3.dist-info
-rw-r--r--  1 username username    0 Sep 16 13:34 pip-21.2.3.virtualenv
drwxr-xr-x  6 username username   86 Sep 16 13:34 pkg_resources
drwxr-xr-x  2 username username   40 Sep 16 13:34 __pycache__
drwxr-xr-x  7 username username 4096 Sep 16 13:34 setuptools
drwxr-xr-x  2 username username  143 Sep 16 13:34 setuptools-58.0.4.dist-info
-rw-r--r--  1 username username   18 Sep 16 13:34 _virtualenv.pth
-rw-r--r--  1 username username 5662 Sep 16 13:34 _virtualenv.py
drwxr-xr-x  4 username username  190 Sep 16 13:34 wheel
drwxr-xr-x  2 username username  130 Sep 16 13:34 wheel-0.37.0.dist-info
-rw-r--r--  1 username username    0 Sep 16 13:34 wheel-0.37.0.virtualenv
(pyb-3.9.7) [username@hostname pybuilder]$ virtualenv test-env
created virtual environment CPython3.9.7.final.0-64 in 128ms
  creator CPython3Posix(dest=/home/username/src/pybuilder/test-env, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/username/.local/share/virtualenv)
    added seed packages: pip==21.2.3, setuptools==57.4.0, setuptools==58.0.4, wheel==0.37.0
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator
(pyb-3.9.7) [username@hostname pybuilder]$ ls -la test-env/lib/python3.9/site-packages/
total 24
drwxr-xr-x 12 username username 4096 Sep 16 13:35 .
drwxr-xr-x  3 username username   27 Sep 16 13:34 ..
drwxr-xr-x  2 username username   44 Sep 16 13:35 _distutils_hack
-rw-r--r--  1 username username  152 Sep 16 13:35 distutils-precedence.pth
drwxr-xr-x  4 username username   92 Sep 16 13:35 pip
drwxr-xr-x  2 username username  130 Sep 16 13:35 pip-21.2.3.dist-info
-rw-r--r--  1 username username    0 Sep 16 13:35 pip-21.2.3.virtualenv
drwxr-xr-x  5 username username   67 Sep 16 13:35 pkg_resources
drwxr-xr-x  2 username username   40 Sep 16 13:34 __pycache__

********  <=======================
drwxr-xr-x  6 username username 4096 Sep 16 13:35 setuptools
drwxr-xr-x  2 username username  126 Sep 16 13:35 setuptools-57.4.0.dist-info
-rw-r--r--  1 username username    0 Sep 16 13:35 setuptools-57.4.0.virtualenv
drwxr-xr-x  2 username username  143 Sep 16 13:34 setuptools-58.0.4.dist-info
*********

-rw-r--r--  1 username username   18 Sep 16 13:35 _virtualenv.pth
-rw-r--r--  1 username username 5662 Sep 16 13:35 _virtualenv.py
drwxr-xr-x  4 username username  190 Sep 16 13:35 wheel
drwxr-xr-x  2 username username  130 Sep 16 13:35 wheel-0.37.0.dist-info
-rw-r--r--  1 username username    0 Sep 16 13:35 wheel-0.37.0.virtualenv

This environment is now broken producing pybuilder/pybuilder#795 due to setuptools-57.4.0.dist-info and setuptools-58.0.4.dist-info being present.

[gaborbernat]

It’s a complex project with lots of hard problems that don’t get noticed until someone runs into them. We’re always looking for new contributors (and eventually maintainer) if you have time to work on issues.

[gaborbernat]

This is an open source project, there’s no staff. I’m the only maintainer, and I do that role mostly in my free time for free. As this feels like an edge that impacts few people it’s very low on my priority list.