Poetry can't handle private default source and additional secondary sources

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Windows 10 Enterprise 1909
• Poetry version: 1.1.13
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/Carl132/c9b5a33b5c3067dae28b0a13268b88b2

Issue

I’m currently working to develop a Jinja template to facilitate Machine Learning workflows in my company. I’ve developed a library which serves as an interface to many GCP systems and this is in the package gcp_tools, which sits in our private Gitlab package registry. I need this dependent library to be installed into my Jinja template repo.

In addition to this, I’ve got the constraint of my environment being in a VPC without any internet access, rather there is a proxy that connects us to PyPi.

Therefore, I’ve set up my pyproject.toml to have the following source configuration:

[[tool.poetry.source]]
name = "gcp_tools"
url = "https://<private_gitlab_repo>/api/v4/projects/<project_id>/packages/pypi/simple"
# Have tried with secondary = true here also

[[tool.poetry.source]]
name = "nexus"
url = "https://<private_pypi_proxy>/nexus/repository/pypi-proxy/simple"
default = true

And I’ve got both of my sources listed in the config for poetry too (I’ve set poetry config http-basic.gcp_tools with my credentials to access the private Gitlab package registry, but there shouldn’t be any authentication for the nexus proxy):

root@1f0dab1f1134:/home/theia/project/machine-learning-deployment-template# poetry config --list
cache-dir = "/root/.cache/pypoetry"
experimental.new-installer = true
installer.parallel = true
repositories.gcp_tools.url = "https://<private_gitlab_repo>/api/v4/projects/<project_id>/packages/pypi/simple"
repositories.nexus.url = "https://<private_pypi_proxy>/nexus/repository/pypi-proxy/simple"
virtualenvs.create = true
virtualenvs.in-project = false
virtualenvs.path = "{cache-dir}/virtualenvs"  # /root/.cache/pypoetry/virtualenvs

However, as you can see from my full stacktrace below, the versioning is failing as it’s trying to access pypi.org for the dependent packages for gcp_tools.

I’ve tried to switch around the different sources, and add secondary = true to the gcp_tools source, but to no avail.

Full Stacktrace

root@1f0dab1f1134:/home/theia/project/machine-learning-deployment-template# poetry add 'gcp_tools>=1.1.0' -vvv
Using virtualenv: /root/.cache/pypoetry/virtualenvs/machine-learning-deployment-template-1tOBkp6x-py3.7
gcp_tools: 1 packages found for gcp-tools >=1.1.0

Updating dependencies
Resolving dependencies...
   1: fact: machine-learning-deployment-template is 2.1.0
   1: derived: machine-learning-deployment-template
   1: fact: machine-learning-deployment-template depends on sklearn (^0.0)
   1: fact: machine-learning-deployment-template depends on xgboost (>=0.9.0,<1.0.0;)
   1: fact: machine-learning-deployment-template depends on seaborn (^0.11.2)
   1: fact: machine-learning-deployment-template depends on matplotlib (^3.4.3)
   1: fact: machine-learning-deployment-template depends on tqdm (>=4.61.0,<4.62.0)
   1: fact: machine-learning-deployment-template depends on pycodestyle (>=2.7.0,<2.8.0)
   1: fact: machine-learning-deployment-template depends on flake8 (>=3.9.2,<3.10.0)
   1: fact: machine-learning-deployment-template depends on pycodestyle_magic (>=0.5.0,<0.6.0)
   1: fact: machine-learning-deployment-template depends on copier (^5.1.0)
   1: fact: machine-learning-deployment-template depends on absl-py (>0.9,<0.11;)
   1: fact: machine-learning-deployment-template depends on plumbum (^1.7.2)
   1: fact: machine-learning-deployment-template depends on gcp-tools (>=1.1.0)
   1: selecting machine-learning-deployment-template (2.1.0)
   1: derived: gcp-tools (>=1.1.0)
   1: derived: plumbum (>=1.7.2,<2.0.0)
   1: derived: absl-py (>0.9,<0.11)
   1: derived: copier (>=5.1.0,<6.0.0)
   1: derived: pycodestyle_magic (>=0.5.0,<0.6.0)
   1: derived: flake8 (>=3.9.2,<3.10.0)
   1: derived: pycodestyle (>=2.7.0,<2.8.0)
   1: derived: tqdm (>=4.61.0,<4.62.0)
   1: derived: matplotlib (>=3.4.3,<4.0.0)
   1: derived: seaborn (>=0.11.2,<0.12.0)
   1: derived: xgboost (>=0.9.0,<1.0.0)
   1: derived: sklearn (>=0.0,<0.1)
gcp_tools: 1 packages found for gcp-tools >=1.1.0
   1: fact: gcp-tools (1.1.0) depends on coverage (>=6.3.2,<7.0.0)
   1: fact: gcp-tools (1.1.0) depends on croniter (>=1.3.4,<2.0.0)
   1: fact: gcp-tools (1.1.0) depends on gcsfs (>=2022.2.0,<2023.0.0)
   1: fact: gcp-tools (1.1.0) depends on google-cloud-aiplatform (>=1.11.0,<2.0.0)
   1: fact: gcp-tools (1.1.0) depends on google-cloud-pipeline-components (>=1.0.0,<2.0.0)
   1: fact: gcp-tools (1.1.0) depends on kfp (>=1.8.11,<2.0.0)
   1: fact: gcp-tools (1.1.0) depends on pandas (>=1.1.5,<2.0.0)
   1: fact: gcp-tools (1.1.0) depends on pyarrow (>=7.0.0,<8.0.0)
   1: fact: gcp-tools (1.1.0) depends on pylint (>=2.12.2,<3.0.0)
   1: fact: gcp-tools (1.1.0) depends on pytest (>=7.0.1,<8.0.0)
   1: fact: gcp-tools (1.1.0) depends on pytest-timeout (>=2.1.0,<3.0.0)
   1: fact: gcp-tools (1.1.0) depends on python-gitlab (>=3.2.0,<4.0.0)
   1: fact: gcp-tools (1.1.0) depends on sklearn (>=0.0,<0.1)
   1: selecting gcp-tools (1.1.0)
   1: derived: python-gitlab (>=3.2.0,<4.0.0)
   1: derived: pytest-timeout (>=2.1.0,<3.0.0)
   1: derived: pytest (>=7.0.1,<8.0.0)
   1: derived: pylint (>=2.12.2,<3.0.0)
   1: derived: pyarrow (>=7.0.0,<8.0.0)
   1: derived: pandas (>=1.1.5,<2.0.0)
   1: derived: kfp (>=1.8.11,<2.0.0)
   1: derived: google-cloud-pipeline-components (>=1.0.0,<2.0.0)
   1: derived: google-cloud-aiplatform (>=1.11.0,<2.0.0)
   1: derived: gcsfs (>=2022.2.0,<2023.0.0)
   1: derived: croniter (>=1.3.4,<2.0.0)
   1: derived: coverage (>=6.3.2,<7.0.0)
nexus: 1 packages found for python-gitlab >=3.2.0,<4.0.0
   1: Version solving took 126.794 seconds.
   1: Tried 1 solutions.

  Stack trace:

  24  /opt/conda/lib/python3.7/site-packages/clikit/console_application.py:131 in run
       129│             parsed_args = resolved_command.args
       130│ 
     → 131│             status_code = command.handle(parsed_args, io)
       132│         except KeyboardInterrupt:
       133│             status_code = 1

  23  /opt/conda/lib/python3.7/site-packages/clikit/api/command/command.py:120 in handle
       118│     def handle(self, args, io):  # type: (Args, IO) -> int
       119│         try:
     → 120│             status_code = self._do_handle(args, io)
       121│         except KeyboardInterrupt:
       122│             if io.is_debug():

  22  /opt/conda/lib/python3.7/site-packages/clikit/api/command/command.py:171 in _do_handle
       169│         handler_method = self._config.handler_method
       170│ 
     → 171│         return getattr(handler, handler_method)(args, io, self)
       172│ 
       173│     def __repr__(self):  # type: () -> str

  21  /opt/conda/lib/python3.7/site-packages/cleo/commands/command.py:92 in wrap_handle
        90│         self._command = command
        91│ 
     →  92│         return self.handle()
        93│ 
        94│     def handle(self):  # type: () -> Optional[int]

  20  /opt/conda/lib/python3.7/site-packages/poetry/console/commands/add.py:173 in handle
       171│             self._installer.whitelist([r["name"] for r in requirements])
       172│ 
     → 173│             status = self._installer.run()
       174│         except BaseException:
       175│             # Using BaseException here as some exceptions, eg: KeyboardInterrupt, do not inherit from Exception

  19  /opt/conda/lib/python3.7/site-packages/poetry/installation/installer.py:103 in run
       101│         local_repo = Repository()
       102│ 
     → 103│         return self._do_install(local_repo)
       104│ 
       105│     def dry_run(self, dry_run=True):  # type: (bool) -> Installer

  18  /opt/conda/lib/python3.7/site-packages/poetry/installation/installer.py:235 in _do_install
       233│             )
       234│ 
     → 235│             ops = solver.solve(use_latest=self._whitelist)
       236│         else:
       237│             self._io.write_line("Installing dependencies from lock file")

  17  /opt/conda/lib/python3.7/site-packages/poetry/puzzle/solver.py:65 in solve
        63│         with self._provider.progress():
        64│             start = time.time()
     →  65│             packages, depths = self._solve(use_latest=use_latest)
        66│             end = time.time()
        67│ 

  16  /opt/conda/lib/python3.7/site-packages/poetry/puzzle/solver.py:234 in _solve
       232│         try:
       233│             result = resolve_version(
     → 234│                 self._package, self._provider, locked=locked, use_latest=use_latest
       235│             )
       236│ 

  15  /opt/conda/lib/python3.7/site-packages/poetry/mixology/__init__.py:7 in resolve_version
       5│     solver = VersionSolver(root, provider, locked=locked, use_latest=use_latest)
       6│ 
     → 7│     return solver.solve()
       8│ 

  14  /opt/conda/lib/python3.7/site-packages/poetry/mixology/version_solver.py:84 in solve
        82│             while next is not None:
        83│                 self._propagate(next)
     →  84│                 next = self._choose_package_version()
        85│ 
        86│             return self._result()

  13  /opt/conda/lib/python3.7/site-packages/poetry/mixology/version_solver.py:372 in _choose_package_version
       370│             dependency = unsatisfied[0]
       371│         else:
     → 372│             dependency = min(*unsatisfied, key=_get_min)
       373│ 
       374│         locked = self._get_locked(dependency)

  12  /opt/conda/lib/python3.7/site-packages/poetry/mixology/version_solver.py:364 in _get_min
       362│                 return (
       363│                     not dependency.marker.is_any(),
     → 364│                     len(self._provider.search_for(dependency)),
       365│                 )
       366│             except ValueError:

  11  /opt/conda/lib/python3.7/site-packages/poetry/puzzle/provider.py:139 in search_for
       137│             packages = self.search_for_url(dependency)
       138│         else:
     → 139│             packages = self._pool.find_packages(dependency)
       140│ 
       141│             packages.sort(

  10  /opt/conda/lib/python3.7/site-packages/poetry/repositories/pool.py:170 in find_packages
       168│         packages = []
       169│         for repo in self._repositories:
     → 170│             packages += repo.find_packages(dependency)
       171│ 
       172│         return packages

   9  /opt/conda/lib/python3.7/site-packages/poetry/repositories/legacy_repository.py:264 in find_packages
       262│             versions = self._cache.store("matches").get(key)
       263│         else:
     → 264│             page = self._get("/{}/".format(dependency.name.replace(".", "-")))
       265│             if page is None:
       266│                 return []

   8  /opt/conda/lib/python3.7/site-packages/poetry/repositories/legacy_repository.py:388 in _get
       386│         url = self._url + endpoint
       387│         try:
     → 388│             response = self.session.get(url)
       389│             if response.status_code == 404:
       390│                 return

   7  /opt/conda/lib/python3.7/site-packages/requests/sessions.py:555 in get
       553│ 
       554│         kwargs.setdefault('allow_redirects', True)
     → 555│         return self.request('GET', url, **kwargs)
       556│ 
       557│     def options(self, url, **kwargs):

   6  /opt/conda/lib/python3.7/site-packages/requests/sessions.py:542 in request
       540│         }
       541│         send_kwargs.update(settings)
     → 542│         resp = self.send(prep, **send_kwargs)
       543│ 
       544│         return resp

   5  /opt/conda/lib/python3.7/site-packages/requests/sessions.py:677 in send
       675│             # Redirect resolving generator.
       676│             gen = self.resolve_redirects(r, request, **kwargs)
     → 677│             history = [resp for resp in gen]
       678│         else:
       679│             history = []

   4  /opt/conda/lib/python3.7/site-packages/requests/sessions.py:677 in <listcomp>
       675│             # Redirect resolving generator.
       676│             gen = self.resolve_redirects(r, request, **kwargs)
     → 677│             history = [resp for resp in gen]
       678│         else:
       679│             history = []

   3  /opt/conda/lib/python3.7/site-packages/requests/sessions.py:245 in resolve_redirects
       243│                     proxies=proxies,
       244│                     allow_redirects=False,
     → 245│                     **adapter_kwargs
       246│                 )
       247│ 

   2  /opt/conda/lib/python3.7/site-packages/requests/sessions.py:655 in send
       653│ 
       654│         # Send the request
     → 655│         r = adapter.send(request, **kwargs)
       656│ 
       657│         # Total elapsed time of the request (approximately)

   1  /opt/conda/lib/python3.7/site-packages/cachecontrol/adapter.py:57 in send
        55│             request.headers.update(self.controller.conditional_headers(request))
        56│ 
     →  57│         resp = super(CacheControlAdapter, self).send(request, **kw)
        58│ 
        59│         return resp

  ConnectionError

  HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/python-gitlab/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f202de8e890>: Failed to establish a new connection: [Errno 110] Connection timed out'))

  at /opt/conda/lib/python3.7/site-packages/requests/adapters.py:516 in send
      512│             if isinstance(e.reason, _SSLError):
      513│                 # This branch is for urllib3 v1.22 and later.
      514│                 raise SSLError(e, request=request)
      515│ 
    → 516│             raise ConnectionError(e, request=request)
      517│ 
      518│         except ClosedPoolError as e:
      519│             raise ConnectionError(e, request=request)
      520│

[goodmami]

(@Carl132 It looks like you might have unredacted tokens and URLs in the first code block of your second post.)

(P.S. too bad GitHub doesn’t have a PM feature)

[neersighted]

That is by-design, currently [[tool.poetry.source]] does not influence the build-system.