Up-to-date packages are re-installed if virtualenvs.create is false

Re-opening #4254, which was closed wrongly, #4329 does not fix this bug.

• I am on the latest current master (22c3aaddc20ad74d3633738093f685fd6dbb998a) Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Ubuntu 20.04

• Poetry version: current master (22c3aaddc20ad74d3633738093f685fd6dbb998a) (also tested with version 1.2.0a2)

• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/jstriebel/6e804c769e2075be390496e68474a36c

Issue

Packages that were already installed (by poetry itself) are re-installed if virtualenvs.create is false. This bug only appears on v1.2, it does not occur on the 1.1 branch or the latest v1.1 releases. I added a reproducible case via a Dockerfile in the linked gist.

When running docker build . on those files, this produces:

Step 12/14 : RUN poetry install
 ---> Running in 73dfd5dcf683
Skipping virtualenv creation, as specified in config file.
Installing dependencies from lock file

Package operations: 3 installs, 0 updates, 2 removals

  • Removing importlib-metadata (4.6.1)
  • Removing zipp (3.5.0)
  • Installing attrs (21.2.0)
  • Installing pyrsistent (0.18.0)
  • Installing jsonschema (3.2.0)
Removing intermediate container 73dfd5dcf683
 ---> 04e982ae2043
Step 13/14 : RUN echo SECOND INSTALL!!!
 ---> Running in 7ec46c74fa47
SECOND INSTALL!!!
Removing intermediate container 7ec46c74fa47
 ---> eff9efcdee5e
Step 14/14 : RUN poetry install
 ---> Running in a877dc9c4864
Skipping virtualenv creation, as specified in config file.
Installing dependencies from lock file

Package operations: 3 installs, 0 updates, 0 removals

  • Installing attrs (21.2.0)
  • Installing pyrsistent (0.18.0)
  • Installing jsonschema (3.2.0)
Removing intermediate container a877dc9c4864

The packages are not installed twice if virtualenvs.create is true (tested by commenting out the COPY poetry.toml / line).

Possibly related previous issues: #1711, #1882 (solved in previous releases, but re-introduced in v1.2)

Given that this was broken multiple times in different ways now, and was not fixed by #4329 as intended, would it be possible to add some tests for this case as well? I’d be happy to help here, but have no experience with the poetry code-base.

[abn]

@jstriebel the root cause here is the sys.path value in ubuntu image’s python installation. The sequence of events that cause this issue is the following:

1. When Poetry discovers installed packages, the sys.path output is used when the system environment is in use. This on your base image returns without /usr/lib/python3.8/site-packages (as debian systems rely on dist-packages for system site. However the local (/usr/local/lib) site is correctly present.

root@413fb0865efc:/# python3 -m site
sys.path = [
    '/',
    '/usr/lib/python38.zip',
    '/usr/lib/python3.8',
    '/usr/lib/python3.8/lib-dynload',
    '/root/.local/lib/python3.8/site-packages',
    '/usr/local/lib/python3.8/dist-packages',
    '/usr/lib/python3/dist-packages',
]
USER_BASE: '/root/.local' (exists)
USER_SITE: '/root/.local/lib/python3.8/site-packages' (exists)
ENABLE_USER_SITE: True

1. Poetry proceeds to download the wheel and install it using pip with the option --prefix /usr.
2. pip installs package to /usr/lib/python3.8/site-packages as per Python spec.
3. When Poetry attempts to detect installed packages again, as the site-packages directory is not in sys.path it is detected as no installed.

You can work around this issue by setting PYTHONPATH.

ENV PYTHONPATH=/usr/lib/python3.8/site-packages

Poetry could try passing --pre /usr/local instead to pip. However, I am not certain there is a realiable way to do this across distros.

Personally, I would always recommend using a virtual environment, even in containers. A lot less volatility.

Additionally, you can verify that this scenario works without issue in another image like shown here.

podman run --rm -i --entrypoint bash python:3.10 <<EOF
set -xe
python -m pip install -q git+https://github.com/python-poetry/poetry.git@master
poetry config virtualenvs.create false
poetry new foobar
pushd foobar
poetry add pycowsay
poetry install
poetry install
EOF

+ python -m pip install git+https://github.com/python-poetry/poetry.git@master
+ poetry config virtualenvs.create false
+ poetry new foobar
Created package foobar in foobar
+ pushd foobar
/foobar /
+ poetry add pycowsay
Skipping virtualenv creation, as specified in config file.
Using version ^0.0.0.1 for pycowsay

Updating dependencies
Resolving dependencies...

Writing lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing pycowsay (0.0.0.1)
+ poetry install
Skipping virtualenv creation, as specified in config file.
Installing dependencies from lock file

No dependencies to install or update

Installing the current project: foobar (0.1.0)
+ poetry install
Skipping virtualenv creation, as specified in config file.
Installing dependencies from lock file

No dependencies to install or update

Installing the current project: foobar (0.1.0)

[jstriebel]

@abn Thanks a lot for the detailed answer, and sorry for my late reply.

I agree, that this seems to be a bug in the configuration. Should sys.prefix rather point to /usr/local, where pip-installed packages are expected under debian/ubuntu? I assume the /usr prefix should only be used for system-packages, e.g. installed via apt.

Poetry could try passing --pre /usr/local instead to pip. However, I am not certain there is a realiable way to do this across distros.

How about simply not providing --prefix to pip? I guess this might be a more stable strategy in the case of no virtualenv? This would ensure that the packages are installed “the standard way”. However, finding the correct installation path is more of a hassle then.

However the local (/usr/local/lib) site is correctly present.

Only the dist-packages directory, not for site-packages, right?

I’m very grateful that you are helping to deal with the package installation chaos! Thanks a ton for your work 🙏

The only “fix”/“workaround” I could see atm is removing the prefix. If that’s something you’d like to avoid, please feel free to close this issue.