Private Repository – Extras suddenly failing to match

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Mac OS Catalina (10.15.5)
• Poetry version: 1.0.9
• Link of a Gist with the contents of your pyproject.toml file:

Issue

I’m encountering a bizarre, frustrating issue when using a package in a private repository that also provides python “extras” options. There are two packages at play here (names de-identified): core and api. The api package depends on the core package.

Previously, the core package was built with the following (simplified) setup.py script:

install_requirements = ['pandas==0.24.2', 'numpy==1.16.4', 'requests==2.23.0', 'ujson==1.35', ]
setup_requirements = []
test_requirements = ['pytest-runner==5.2', 'pytest==5.4.1', ]
extra_requirements = {
    'aws': ['s3fs==0.1.5', ],
    'excel': ['XlsxWriter==1.1.1', 'xlrd==1.2.0', ],
    'performance': ['simplejson==3.16.0', ],
    'full': [],
}

extra_requirements['full'] = list(set(itertools.chain.from_iterable(extra_requirements.values())))

setup(
    # <snip>
    name='core',
    python_requires='>=3, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, <4',
    install_requires=install_requirements,
    extras_require=extra_requirements,
    license="proprietary",
    include_package_data=True,
    setup_requires=setup_requirements,
    tests_require=test_requirements,
    version='1.0.0',
    # <snip>
)

When we migrated to poetry, the core library shipped with the following (simplified) pyproject.toml:

[tool.poetry]
name = "core"
version = "2.0.0"
# <snip>

[tool.poetry.dependencies]
python = "^3.7"

pandas = "0.24.2"
numpy = "1.16.4"
requests = "2.23.0"
ujson = "1.35"

s3fs = {version = "0.1.5", optional = true}
boto3 = {version = "1.14.1", optional = true}
XlsxWriter = {version = "1.1.1", optional = true}
xlrd = {version = "1.2.0", optional = true}
simplejson = {version = "3.16.0", optional = true}

[tool.poetry.extras]
aws = ["s3fs", "boto3"]
excel = ["XlsxWriter", "xlrd"]
performance = ["simplejson"]
full = ["s3fs", "XlsxWriter", "xlrd", "simplejson"]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

For simplicity, let’s say core went from 1.0.0 to 2.0.0 with the poetry changes.

Next, the api package attempts to install the core package via its pyproject.toml:

[tool.poetry]
name = "api"
# <snip>

[tool.poetry.dependencies]
python = "^3.7"

core = {version = "1.0.0", extras = ["full"]}

[[tool.poetry.source]]
name = "private"
url = "https://private-repository/pypi/core/simple/"

Running poetry update with core@1.0.0 works as expected, with the extra packages listed in the [full] requirements being properly installed:

Updating dependencies
Resolving dependencies... (15.6s)

Writing lock file


Package operations: 4 installs, 1 update, 0 removals

  - Installing s3fs (0.1.5)
  - Installing simplejson (3.16.0)
  - Installing xlrd (1.2.0)
  - Installing xlsxwriter (1.1.1)
  - Installing core (1.0.0)

However, changing the version tag to 2.0.0 – the version with poetry – causes some bizarre behavior. The first time I run poetry update with core@2.0.0, the [full] packages are removed from the lock file.

Updating dependencies
Resolving dependencies... (15.8s)

Writing lock file


Package operations: 0 installs, 1 update, 4 removals

  - Updating core (1.0.0 -> 2.0.0)
  - Removing s3fs (0.1.5)
  - Removing simplejson (3.16.0)
  - Removing xlrd (1.2.0)
  - Removing xlsxwriter (1.1.1)

The second (and subsequent) time I run poetry update (no changes), I get an error:

Updating dependencies
Resolving dependencies... (15.1s)

[SolverProblemError]
Because core (2.0.0) depends on xlrd (1.2.0) which doesn't match any versions, core is forbidden.
So, because api depends on core (2.0.0), version solving failed.

My guess is that poetry is trying to install the [full] extras from the private repository with 2.0.0 and fails to fall back to PyPI for some reason, which is why it fails to find it… although I don’t know why it removes the extras in the first place.

I’m pretty stuck here, so I would appreciate some guidance.

[seansfkelley]

I’m seeing this issue too: private repository and dependency extras being both required by the solver, but also ignored by the solver. The issue seems to be related to overlapping extras, as we were able to solve it by taking extras blocks that looked like this:

[tool.poetry.extras]
server = [
  "sqlalchemy",
  "fastapi",
  "uvicorn",
]
executor = [
  "fastapi",
  "uvicorn",
  "typer",
]

and split their overlaps so it looked like this:

[tool.poetry.extras]
server = [
  "sqlalchemy",
]
executor = [
  "typer",
]
server_or_executor = [
  "fastapi",
  "uvicorn",
]

(This was a wild crapshoot guess – I happened to notice that the two dependencies giving us issues were the only ones who appeared in the published PKG-INFO with the word “or”, so at first I thought it was some kind of dependency specification syntax error.)

If you take this far enough, then every dependency will boil down to its own extra, which is really not great. That said, this workaround seems stable enough for the time being.

The issue is compounded by error reporting that is inconsistent and therefore misleading. An excerpt follows in heavily redacted form below (note that this is before the splitting of the overlaps described above):

(test) ➜  test poetry add my-private-dependency -E executor -vvv
Using virtualenv: /Users/<redacted>/.pyenv/versions/3.8.2/envs/test
PyPI: No packages found for my-private-dependency *
private-pypi: 8 packages found for my-private-dependency *
Using version ^0.8.0 for my-private-dependency

Updating dependencies
Resolving dependencies...
   1: fact: root-pkg is 2.0.0
   1: derived: root-pkg
   1: fact: root-pkg depends on my-private-dependency (^0.8.0)
   1: selecting root-pkg (2.0.0)
   1: derived: my-private-dependency (^0.8.0)
PyPI: No packages found for my-private-dependency >=0.8.0,<0.9.0
private-pypi: 1 packages found for my-private-dependency >=0.8.0,<0.9.0
PyPI: Getting info for my-private-dependency (0.8.0) from PyPI
   1: fact: my-private-dependency (0.8.0) depends on fastapi (>=0.54.1,<0.55.0)
   1: fact: my-private-dependency (0.8.0) depends on typer (>=0.3.0,<0.4.0)
   1: fact: my-private-dependency (0.8.0) depends on uvicorn (>=0.11.5,<0.12.0)
   1: selecting my-private-dependency (0.8.0)
   1: derived: uvicorn (>=0.11.5,<0.12.0)
   1: derived: typer (>=0.3.0,<0.4.0)
   1: derived: fastapi (>=0.54.1,<0.55.0)
PyPI: 2 packages found for uvicorn >=0.11.5,<0.12.0
private-pypi: 2 packages found for uvicorn >=0.11.5,<0.12.0
PyPI: 2 packages found for fastapi >=0.54.1,<0.55.0
private-pypi: 2 packages found for fastapi >=0.54.1,<0.55.0
   1: fact: typer (0.3.0) depends on click (>=7.1.1,<7.2.0)
   1: selecting typer (0.3.0)
   1: derived: click (>=7.1.1,<7.2.0)
   1: selecting uvicorn (0.11.6)
   1: selecting fastapi (0.54.2)
   1: Version solving took 1.684 seconds.
   1: Tried 1 solutions.

[SolverProblemError]
Because no versions of my-private-dependency match >0.8.0,<0.9.0
 and my-private-dependency (0.8.0) depends on uvicorn (>=0.11.5,<0.12.0), my-private-dependency (>=0.8.0,<0.9.0) requires uvicorn (>=0.11.5,<0.12.0).
So, because no versions of uvicorn match >=0.11.5,<0.12.0
 and root-pkg depends on my-private-dependency (^0.8.0), version solving failed.

Notice that it correctly reports that PyPI and the private PyPI instance both have two matching versions of uvicorn (which is true), and that it even later selects* what would be a compatible version of uvicorn (0.11.6), before complaining that there is no legal solution.

* I assume “select” means “attempt to solve for” rather than “decide is the correct solution”.

[SamuelMS]

Closing this as we went with a different solution.