"Unable to find installation candidates for" packages cached from a LegacyRepository (`/simple`)

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

  • Caveat : I have insufficient detail to decide if #4679 is a dupe of this issue, but it is possible since it has similar symptoms

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• CodeBuild image aws/codebuild/standard:5.0 (Ubuntu 20.04)

• Poetry 1.1.11 : as installed by pip

  • Yes, this is not the recommended way to install Poetry
  • However, the recommended way ought to have the same issue (installs same version of cachecontrol, at present)
  • Confirmed that using install-poetry.py on a fresh install reproduces this issue

Issue

Have been debugging this inside a container based on aws/codebuild/standard:5.0 (since this is the target build container for our CI pipelines).

Pipelines are installing poetry via pip and failing. Build works fine on my workstation, even after a poetry self update, which doesn’t seem to update dependencies if they already meet the version constraints (like pip).

After a fresh install of Poetry 1.1.11 via install-poetry.py, now also having similar issues (with different symptoms but I presume the same cause) locally.

Issues are resolved by activating Poetry’s venv and running

pip install cachecontrol==0.12.6

This only applies to packages installed (from a Nexus repo) via the /simple interface, which uses the LegacyRepository class.

Suspected issue

Working on a deeper diagnosis, but thought I’d get the bug report opened.

cachecontrol released 0.12.7 as of 12 hours ago, clearly there is a bug or change in behaviour (aka - bug, for a patch version bump) that breaks Poetry for this use case.

The issue seems to be caching of the HTML metadata from the /simple type repo ; while the cache file on disk is successfully updated each time it is fetched, the cache-wrapped response object for the metadata returns an empty byte buffer to the program. This results in the Page object not finding any links and get_links_for_package failing.

In a tree with a lockfile and in an environment with an empty cache, this means you will be able to install that package once and once only - the first time caches the metadata, which then subsequently triggers this bug and prevents further access to the metadata. Sadly all my pipelines build multiple packages with common dependencies.

Now going to look and see what the problem in cachecontrol is.

Might I humbly suggest that as a tool which promotes stable builds … that Poetry use some of it’s own medicine to install itself, and use a lockfile or output of pip freeze to create a reproducible install for a given version. This is definitely not the first occasion on which a reproducible installation would have prevented issues - see #4523 and possibly also #4528

Terminal output

root@d829437ec7b5:/home/src/python/account# poetry install -vvv
Debug wait
Debug attached
Using virtualenv: /root/.cache/pypoetry/virtualenvs/account-D8L0MJuz-py3.9
Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 45 installs, 0 updates, 0 removals, 2 skipped

  • Removing importlib-metadata (4.8.1): Pending...
  • Removing importlib-metadata (4.8.1): Skipped for the following reason: Not currently installed
  • Removing zipp (3.5.1): Pending...
  • Removing zipp (3.5.1): Skipped for the following reason: Not currently installed
  • Installing six (1.16.0): Pending...
  • Installing six (1.16.0): Failed

  RuntimeError

  Unable to find installation candidates for six (1.16.0)

  at ~/.pyenv/versions/3.9.1/lib/python3.9/site-packages/poetry/installation/chooser.py:72 in choose_for
       68│ 
       69│             links.append(link)
       70│ 
       71│         if not links:
    →  72│             raise RuntimeError(
       73│                 "Unable to find installation candidates for {}".format(package)
       74│             )
       75│ 
       76│         # Get the best link

also

❯ poetry update
Updating dependencies
Resolving dependencies... (0.8s)

  PackageNotFound

  Package six (1.16.0) not found.

  at ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/repositories/pool.py:149 in package
      145│                     self._packages.append(package)
      146│ 
      147│                     return package
      148│ 
    → 149│         raise PackageNotFound("Package {} ({}) not found.".format(name, version))
      150│ 
      151│     def find_packages(
      152│         self, dependency,
      153│     ):

[f0ff886f]

Thank you for finding this out so quickly, we were just in the middle of triaging our CI failing to build anything. I can confirm on Py 3.9 that pinning cachecontrol==0.12.6 helps, however it isn’t enough for Py 3.6 and 3.7 (but Py 3.8 and 3.9 were fine), so cachecontrol[filecache]==0.12.6 was enough to fix it across all versions.

Might I humbly suggest that as a tool which promotes stable builds … that Poetry use some of it’s own medicine to install itself, and use a lockfile or output of pip freeze to create a reproducible install for a given version.

💯

[awilkins]

@numine777 Not seeing this problem recurring ; I’m building multi-project python build jobs in CodeBuild that reliably triggered this cache problem, and I’m installing Poetry only via pip (the script essentially makes a special venv and then installs via pip). Do you have any stale caches hanging around? I recommend you clear them if so.

cachecontrol 0.12.10 only seems to amend the package metadata surrounding licenses to make certain license auditing use cases work better, no functional changes that ought to have broken it again.

If you like, maybe try this alternative install script : this is the normal install-poetry.sh script, but with a fixed version of every dependency baked into it. It will only install version 1.1.11 at present.