Regression: 403 Auth Failure for S3-hosted (Cloudfront) Private Repos

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate. (There are similar tickets, notably #3291 but they aren’t specifically targeting a repo hosted on S3/Cloudfront)

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: python:3.7-alpine (Docker)

• Poetry version: 1.1.4

• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/dvf/a117ebdb0358cb388c5145cfdd39b46e

Issue

• We noticed a regression from Poetry 1.0.3 to 1.1.4.
• We’re using a private PyPI repository hosted on S3 protected by Cloudfront using Basic Auth (we use s3pypi)

When we tried to install a new private dependency, or remove a non-private dependency (with common sub dependencies with a private dependency) we saw this error:

➜  project git:(master) poetry remove django-structlog
Updating dependencies
Resolving dependencies... (0.2s)

  RepositoryError

  403 Client Error: Forbidden for url: https://XXXXXXXXXXXXXXXX/pyjwt/

  at ~/.poetry/lib/poetry/repositories/legacy_repository.py:393 in _get
      389│             if response.status_code == 404:
      390│                 return
      391│             response.raise_for_status()
      392│         except requests.HTTPError as e:
    → 393│             raise RepositoryError(e)
      394│ 
      395│         if response.status_code in (401, 403):
      396│             self._log(
      397│                 "Authorization error accessing {url}".format(url=url), level="warn"

Hypothesis of why this is happening

By default, Cloudfront returns a 403 for non-existing files. This is a good security practice as it prevents leakage of information to unauthenticated parties. However, (from the above code) Poetry only falls back to public PyPI if it encounters a 404 when requesting a package.

Quick fix

The quick fix for this is to have your private repo (in our case Cloudfront) return a default error response code of 404. Here are docs on how to do that: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/custom-error-pages-response-code.html

Recommendations

Poetry shouldn’t fall back to the public repo on a 401/403/404 without first checking if it can successfully auth to the repo. And failing that, it should throw an appropriate error.

Thanks to @proxyroot and @vagelim for helping debug and fix.

[jeremysprofile]

This also affects Sonatype Nexus PyPI repositories when using poetry 1.1.4, and downgrading to 1.0.10 fixes the issue there as well.

[cereblanco]

hello! if no one started working on this yet, I can create the fix. 💭