Further SSL problems with Poetry and a private Gitlab package registry

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: macOS 11.3
• Poetry version: 1.1.6

Issue

Closely related to #745 and #1012, I am experiencing problems working with poetry against a private (Gitlab) based package registry, on an internal project. The locked nature of this make this very hard for me to do a public replication of the problem, so apologies whilst I have to be descriptive (if a little vague for some sensitive info).

I have two poetry based python+django projects. The first (atlas-models) I have managed to package as a reusable app, and build and publish (with just poetry) to our internal Gitlab Package Registry. Presently there are two versions of the package in the registry (but I’ve tried with between 1 & 3 versions there, that doesn’t appear to be a factor).

In my second project, when I attempt to add the first package via poetry add atlas-models -vvv, I get the following (abbreviated) output:

➜  ouh-mpages-development-repo git:(master) ✗ poetry add atlas-models -vvv

Using virtualenv: /Users/carl/Projects/pages.oxnet/documentation/ouh-mpages-development/ouh-mpages-development-repo/.venv
pages-oxnet-atlas: 2 packages found for atlas-models *
PyPI: No packages found for atlas-models *
Using version ^0.3.1 for atlas-models

Updating dependencies
Resolving dependencies...
   1: fact: mpages mkdocs build is 0.1.0
   1: derived: mpages mkdocs build
   1: fact: mpages mkdocs build depends on Django (^3.2)
   1: fact: mpages mkdocs build depends on atlas-models (^0.3.1)
   1: selecting mpages mkdocs build (0.1.0)
   1: derived: atlas-models (>=0.3.1,<0.4.0)
   1: derived: Django (>=3.2,<4.0)
pages-oxnet-atlas: 1 packages found for atlas-models >=0.3.1,<0.4.0
PyPI: No packages found for atlas-models >=0.3.1,<0.4.0
   1: fact: atlas-models (0.3.1) depends on Django (>=3.2,<4.0)
   1: fact: atlas-models (0.3.1) depends on cx-oracle (>=8,<9)
   1: selecting atlas-models (0.3.1)
   1: derived: cx-oracle (>=8,<9)
   1: fact: django (3.2) depends on asgiref (>=3.3.2,<4)
   1: fact: django (3.2) depends on pytz (*)
   1: fact: django (3.2) depends on sqlparse (>=0.2.2)
   1: selecting django (3.2)
   1: derived: sqlparse (>=0.2.2)
   1: derived: pytz
   1: derived: asgiref (>=3.3.2,<4)
   1: selecting cx-oracle (8.1.0)
   1: selecting sqlparse (0.4.1)
   1: selecting pytz (2021.1)
   1: selecting asgiref (3.3.4)
   1: Version solving took 0.723 seconds.
   1: Tried 1 solutions.

Finding the necessary packages for the current system

Package operations: 1 install, 0 updates, 0 removals, 5 skipped

  • Installing asgiref (3.3.4): Pending...
  • Installing asgiref (3.3.4): Skipped for the following reason: Already installed
  • Installing pytz (2021.1): Pending...
  • Installing pytz (2021.1): Skipped for the following reason: Already installed
  • Installing sqlparse (0.4.1): Pending...
  • Installing sqlparse (0.4.1): Skipped for the following reason: Already installed
  • Installing cx-oracle (8.1.0): Pending...
  • Installing cx-oracle (8.1.0): Skipped for the following reason: Already installed
  • Installing django (3.2): Pending...
  • Installing django (3.2): Skipped for the following reason: Already installed
  • Installing atlas-models (0.3.1): Pending...
Retrying HTTP request in 0.5 seconds.
Retrying HTTP request in 1.0 seconds.
Retrying HTTP request in 1.5 seconds.
Retrying HTTP request in 2.0 seconds.
Retrying HTTP request in 2.5 seconds.
  • Installing atlas-models (0.3.1): Failed

  SSLError

  HTTPSConnectionPool(host='oxnetcnsc01.oxnet.nhs.uk', port=443): Max retries exceeded with url: /api/v4/projects/202/packages/pypi/files/b278cb68cacfff2d3fd4a3d1a82c6ff12dc192c332e67db76cbdf03a02a10c2a/atlas_models-0.3.1-py3-none-any.whl (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)')))

  at ~/.poetry/lib/poetry/_vendor/py3.9/requests/adapters.py:514 in send
      510│                 raise ProxyError(e, request=request)
      511│ 
      512│             if isinstance(e.reason, _SSLError):
      513│                 # This branch is for urllib3 v1.22 and later.
    → 514│                 raise SSLError(e, request=request)
      515│ 
      516│             raise ConnectionError(e, request=request)
      517│ 
      518│         except ClosedPoolError as e:


Failed to add packages, reverting the pyproject.toml file to its original content.

The pyproject.toml reads as:

name = "MPages MkDocs Build"
version = "0.1.0"
description = "Python / Django application *snip*"
authors = ["Carl Marshall <email-redacted>"]
license = "Apache-2.0"

[tool.poetry.dependencies]
python = "^3.9"
Django = "^3.2"

[tool.poetry.dev-dependencies]

[[tool.poetry.source]]
name = "pages-oxnet-atlas"
url = "https://oxnetcnsc01.oxnet.nhs.uk/api/v4/projects/202/packages/pypi/simple"
secondary = true

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Additionally, poetry config --list -vvv results in:

Loading configuration file /Users/carl/Library/Application Support/pypoetry/config.toml
Loading configuration file /Users/carl/Library/Application Support/pypoetry/auth.toml
cache-dir = "/Users/carl/Library/Caches/pypoetry"
certificates.oxnetcnsc01.cert = "/Users/carl/Projects/pages.oxnet/oxnet-combined.pem"  # None
certificates.pages-oxnet.cert = "/Users/carl/Projects/pages.oxnet/oxnet-combined.pem"  # None
certificates.pages-oxnet-atlas.cert = "/Users/carl/Projects/pages.oxnet/oxnet-combined.pem"  # None
experimental.new-installer = true
installer.parallel = true
repositories.pages-oxnet.url = "https://oxnetcnsc01.oxnet.nhs.uk/api/v4/projects/202/packages/pypi"
virtualenvs.create = true
virtualenvs.in-project = true
virtualenvs.path = "{cache-dir}/virtualenvs"  # /Users/carl/Library/Caches/pypoetry/virtualenvs

I have also done poetry config http-basic.pages-oxnet user.name token-code for both .pages-oxnet and .pages-oxnet-atlas. This was all pretty much put in place to get the build and publish workflows to succeed. I also added certificates.oxnetcnsc01.cert and http-basic.oxnetcnsc01 as well to ensure I hadn’t overlooked anything.

I can see that the process is locating the latest version of the atlas-models package from the gitlab private registry, but is clearly having a problem with downloading the wheel file.

The SSL certificates you can see in the config output above are seemingly valid as they resolved the publish SSL connection issues.

If I try to set the environment variable though (to catch a potential issue with requests as suggested in an earlier bug report), I get a whole host of different errors, i.e.:

• export REQUESTS_CA_BUNDLE=/Users/carl/Projects/pages.oxnet/oxnet-combined.pem followed by
• poetry add atlas-models -vvv results in a much faster set of errors:

Using virtualenv: /Users/carl/Projects/pages.oxnet/documentation/ouh-mpages-development/ouh-mpages-development-repo/.venv
pages-oxnet-atlas: 2 packages found for atlas-models *

  Stack trace:

  23  ~/.poetry/lib/poetry/_vendor/py3.9/clikit/console_application.py:131 in run
       129│             parsed_args = resolved_command.args
       130│ 
     → 131│             status_code = command.handle(parsed_args, io)
       132│         except KeyboardInterrupt:
       133│             status_code = 1

  22  ~/.poetry/lib/poetry/_vendor/py3.9/clikit/api/command/command.py:120 in handle
       118│     def handle(self, args, io):  # type: (Args, IO) -> int
       119│         try:
     → 120│             status_code = self._do_handle(args, io)
       121│         except KeyboardInterrupt:
       122│             if io.is_debug():

  21  ~/.poetry/lib/poetry/_vendor/py3.9/clikit/api/command/command.py:171 in _do_handle
       169│         handler_method = self._config.handler_method
       170│ 
     → 171│         return getattr(handler, handler_method)(args, io, self)
       172│ 
       173│     def __repr__(self):  # type: () -> str

  20  ~/.poetry/lib/poetry/_vendor/py3.9/cleo/commands/command.py:92 in wrap_handle
        90│         self._command = command
        91│ 
     →  92│         return self.handle()
        93│ 
        94│     def handle(self):  # type: () -> Optional[int]

  19  ~/.poetry/lib/poetry/console/commands/add.py:106 in handle
       104│             return 0
       105│ 
     → 106│         requirements = self._determine_requirements(
       107│             packages,
       108│             allow_prereleases=self.option("allow-prereleases"),

  18  ~/.poetry/lib/poetry/console/commands/init.py:328 in _determine_requirements
       326│             elif "version" not in requirement:
       327│                 # determine the best version automatically
     → 328│                 name, version = self._find_best_version_for_package(
       329│                     requirement["name"],
       330│                     allow_prereleases=allow_prereleases,

  17  ~/.poetry/lib/poetry/console/commands/init.py:361 in _find_best_version_for_package
       359│ 
       360│         selector = VersionSelector(self._get_pool())
     → 361│         package = selector.find_best_candidate(
       362│             name, required_version, allow_prereleases=allow_prereleases, source=source
       363│         )

  16  ~/.poetry/lib/poetry/version/version_selector.py:32 in find_best_candidate
       30│             },
       31│         )
     → 32│         candidates = self._pool.find_packages(dependency)
       33│         only_prereleases = all([c.version.is_prerelease() for c in candidates])
       34│ 

  15  ~/.poetry/lib/poetry/repositories/pool.py:165 in find_packages
       163│         packages = []
       164│         for repo in self._repositories:
     → 165│             packages += repo.find_packages(dependency)
       166│ 
       167│         return packages

  14  ~/.poetry/lib/poetry/repositories/pypi_repository.py:101 in find_packages
        99│ 
       100│         try:
     → 101│             info = self.get_package_info(dependency.name)
       102│         except PackageNotFound:
       103│             self._log(

  13  ~/.poetry/lib/poetry/repositories/pypi_repository.py:202 in get_package_info
       200│             return self._get_package_info(name)
       201│ 
     → 202│         return self._cache.store("packages").remember_forever(
       203│             name, lambda: self._get_package_info(name)
       204│         )

  12  ~/.poetry/lib/poetry/_vendor/py3.9/cachy/repository.py:174 in remember_forever
       172│             return val
       173│ 
     → 174│         val = value(callback)
       175│ 
       176│         self.forever(key, val)

  11  ~/.poetry/lib/poetry/_vendor/py3.9/cachy/helpers.py:6 in value
       4│ def value(val):
       5│     if callable(val):
     → 6│         return val()
       7│ 
       8│     return val

  10  ~/.poetry/lib/poetry/repositories/pypi_repository.py:203 in <lambda>
       201│ 
       202│         return self._cache.store("packages").remember_forever(
     → 203│             name, lambda: self._get_package_info(name)
       204│         )
       205│ 

   9  ~/.poetry/lib/poetry/repositories/pypi_repository.py:207 in _get_package_info
       205│ 
       206│     def _get_package_info(self, name):  # type: (str) -> dict
     → 207│         data = self._get("pypi/{}/json".format(name))
       208│         if data is None:
       209│             raise PackageNotFound("Package [{}] not found.".format(name))

   8  ~/.poetry/lib/poetry/repositories/pypi_repository.py:315 in _get
       313│     def _get(self, endpoint):  # type: (str) -> Union[dict, None]
       314│         try:
     → 315│             json_response = self.session.get(self._base_url + endpoint)
       316│         except requests.exceptions.TooManyRedirects:
       317│             # Cache control redirect loop.

   7  ~/.poetry/lib/poetry/_vendor/py3.9/requests/sessions.py:555 in get
       553│ 
       554│         kwargs.setdefault('allow_redirects', True)
     → 555│         return self.request('GET', url, **kwargs)
       556│ 
       557│     def options(self, url, **kwargs):

   6  ~/.poetry/lib/poetry/_vendor/py3.9/requests/sessions.py:542 in request
       540│         }
       541│         send_kwargs.update(settings)
     → 542│         resp = self.send(prep, **send_kwargs)
       543│ 
       544│         return resp

   5  ~/.poetry/lib/poetry/_vendor/py3.9/requests/sessions.py:677 in send
       675│             # Redirect resolving generator.
       676│             gen = self.resolve_redirects(r, request, **kwargs)
     → 677│             history = [resp for resp in gen]
       678│         else:
       679│             history = []

   4  ~/.poetry/lib/poetry/_vendor/py3.9/requests/sessions.py:677 in <listcomp>
       675│             # Redirect resolving generator.
       676│             gen = self.resolve_redirects(r, request, **kwargs)
     → 677│             history = [resp for resp in gen]
       678│         else:
       679│             history = []

   3  ~/.poetry/lib/poetry/_vendor/py3.9/requests/sessions.py:237 in resolve_redirects
       235│             else:
       236│ 
     → 237│                 resp = self.send(
       238│                     req,
       239│                     stream=stream,

   2  ~/.poetry/lib/poetry/_vendor/py3.9/requests/sessions.py:655 in send
       653│ 
       654│         # Send the request
     → 655│         r = adapter.send(request, **kwargs)
       656│ 
       657│         # Total elapsed time of the request (approximately)

   1  ~/.poetry/lib/poetry/_vendor/py3.9/cachecontrol/adapter.py:53 in send
        51│             request.headers.update(self.controller.conditional_headers(request))
        52│ 
     →  53│         resp = super(CacheControlAdapter, self).send(request, **kw)
        54│ 
        55│         return resp

  SSLError

  HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/atlas-models/json/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)')))

  at ~/.poetry/lib/poetry/_vendor/py3.9/requests/adapters.py:514 in send
      510│                 raise ProxyError(e, request=request)
      511│ 
      512│             if isinstance(e.reason, _SSLError):
      513│                 # This branch is for urllib3 v1.22 and later.
    → 514│                 raise SSLError(e, request=request)
      515│ 
      516│             raise ConnectionError(e, request=request)
      517│ 
      518│         except ClosedPoolError as e:

When I unset REQUESTS_CA_BUNDLE and retry the add command, we’re back to the original error outputs and timeout.

I am now at a loss of what else to try here. Something appears to be partly broken in the process of adding my package from the internal registry, and this is now stalling my whole development plan. I don’t really want to have to remove poetry from this setup, but I do note there’s very little Gitlab recognition of Poetry usage (none in their own documentation) and most examples I’m finding at either general pip or GitHub solutions.

Please let me know if there’s any other practical tests, or configuration I can attempt, or if this is enough information on which to diagnose a bug.

Thank you for your time and help.

[nylocx]

I hope it is ok to piggyback on this ticket. I have a pretty simple project and a self signed repository. I did the following:

poetry config repositories.myrepo "https://my.repo.tld/simple/"
poetry config certificates.myrepo.cert "/absolute/path/to/my.repo.tld.pem"

Than I added a section in my pyproject.toml

[[tool.poetry.source]]
name = "myrepo"
url = "https://my.repo.tld/simple/"
secondary = true

If I run a poetry add <any-package-that-is-on-pypi> I get:

HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/networkx/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)')))

If I just remove the tool.poetry.source block from my pyproject.toml everything is back to normal and works. I’m on Poetry 1.1.6

It looks like setting any certificate in the config overrides the my system certificate and the workaround from marshalc does not work for me setting SSL_CERT_FILE or REQUESTS_CA_BUNDLE to a valid certificate bundle does not change the behavior. I’m happy for any hint, as this currently blocks my progress.

[R1tschY]

I also had this problem. In the end I fixed it for me with removing the cert repository configuration in the auth.toml:

poetry config certificates.myrepo.cert --unset

Additional I used the fix from marshalc to set the explicit certificate store and added the missing certificate to the CERT_PATH file:

CERT_PATH=$(python -c "import certifi; print(certifi.where())")
echo PATH/TO/CA.crt >> $CERT_PATH
export REQUESTS_CA_BUNDLE=${CERT_PATH}
export SSL_CERT_FILE=${CERT_PATH}