poetry dependency resolution oddity

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: macOS 12.1 arm64
• Poetry version: 1.1.12
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/pravindahal/f6eab4bfac7b342f566cd142ef80443d

Issue

I am having trouble understanding how poetry determines dependencies. The tensorflow-macos package does not seem to depend on tensorflow-io-gcs-filesystem when installed via pip but does depend on it when installed via poetry and I can’t understand why.

I am trying to add tensorflow-macos, like so

poetry add tensorflow-macos --lock
poetry show tensorflow-macos --tree

I get this:

├── absl-py >=0.4.0
│   └── six *
├── astunparse >=1.6.0
│   └── six >=1.6.1,<2.0
├── flatbuffers >=1.12,<3.0
├── gast >=0.2.1,<0.5.0
├── google-pasta >=0.1.1
│   └── six *
├── grpcio >=1.24.3,<2.0
│   └── six >=1.5.2
├── h5py >=2.9.0
│   └── numpy >=1.14.5
├── keras >=2.7.0rc0,<2.8
├── keras-preprocessing >=1.1.1
│   ├── numpy >=1.9.1
│   └── six >=1.9.0
├── libclang >=9.0.1
├── numpy >=1.14.5
├── opt-einsum >=2.3.2
│   └── numpy >=1.7
├── protobuf >=3.9.2
├── six >=1.12.0
├── tensorboard >=2.6,<3.0
│   ├── absl-py >=0.4
│   │   └── six *
│   ├── google-auth >=1.6.3,<3
│   │   ├── cachetools >=2.0.0
│   │   ├── pyasn1-modules >=0.2.1
│   │   │   └── pyasn1 >=0.4.6,<0.5.0
│   │   ├── rsa >=3.1.4
│   │   │   └── pyasn1 >=0.1.3 (circular dependency aborted here)
│   │   └── six >=1.9.0 (circular dependency aborted here)
│   ├── google-auth-oauthlib >=0.4.1,<0.5
│   │   ├── google-auth >=1.0.0 (circular dependency aborted here)
│   │   └── requests-oauthlib >=0.7.0
│   │       ├── oauthlib >=3.0.0
│   │       └── requests >=2.0.0
│   │           ├── certifi >=2017.4.17
│   │           ├── charset-normalizer >=2.0.0,<2.1.0
│   │           ├── idna >=2.5,<4
│   │           └── urllib3 >=1.21.1,<1.27
│   ├── grpcio >=1.24.3
│   │   └── six >=1.5.2 (circular dependency aborted here)
│   ├── markdown >=2.6.8
│   │   └── importlib-metadata >=4.4
│   │       └── zipp >=0.5
│   ├── numpy >=1.12.0
│   ├── protobuf >=3.6.0
│   ├── requests >=2.21.0,<3 (circular dependency aborted here)
│   ├── tensorboard-data-server >=0.6.0,<0.7.0
│   ├── tensorboard-plugin-wit >=1.6.0
│   └── werkzeug >=0.11.15
├── tensorflow-estimator >=2.7.0rc0,<2.8
├── **tensorflow-io-gcs-filesystem** >=0.21.0
├── termcolor >=1.1.0
├── typing-extensions >=3.6.6
└── wrapt >=1.11.0

When I actually install the package via pip, and use pipdeptree to identify the dependencies, I see the following:

poetry run pip install pipdepdree
poetry run pip install tensorflow-macos
poetry run pipdeptree

tensorflow-macos==2.7.0
  - absl-py [required: >=0.4.0, installed: 1.0.0]
    - six [required: Any, installed: 1.15.0]
  - astunparse [required: >=1.6.0, installed: 1.6.3]
    - six [required: >=1.6.1,<2.0, installed: 1.15.0]
    - wheel [required: >=0.23.0,<1.0, installed: 0.37.1]
  - flatbuffers [required: >=1.12,<3.0, installed: 2.0]
  - gast [required: >=0.2.1,<0.5.0, installed: 0.4.0]
  - google-pasta [required: >=0.1.1, installed: 0.2.0]
    - six [required: Any, installed: 1.15.0]
  - grpcio [required: >=1.24.3,<2.0, installed: 1.43.0]
    - six [required: >=1.5.2, installed: 1.15.0]
  - h5py [required: >=2.9.0, installed: 3.6.0]
    - numpy [required: >=1.14.5, installed: 1.22.1]
  - keras [required: >=2.7.0rc0,<2.8, installed: 2.7.0]
  - keras-preprocessing [required: >=1.1.1, installed: 1.1.2]
    - numpy [required: >=1.9.1, installed: 1.22.1]
    - six [required: >=1.9.0, installed: 1.15.0]
  - libclang [required: >=9.0.1, installed: 12.0.0]
  - numpy [required: >=1.14.5, installed: 1.22.1]
  - opt-einsum [required: >=2.3.2, installed: 3.3.0]
    - numpy [required: >=1.7, installed: 1.22.1]
  - protobuf [required: >=3.9.2, installed: 3.19.3]
  - six [required: >=1.12.0, installed: 1.15.0]
  - tensorboard [required: ~=2.6, installed: 2.7.0]
    - absl-py [required: >=0.4, installed: 1.0.0]
      - six [required: Any, installed: 1.15.0]
    - google-auth [required: >=1.6.3,<3, installed: 1.6.3]
      - cachetools [required: >=2.0.0, installed: 5.0.0]
      - pyasn1-modules [required: >=0.2.1, installed: 0.2.8]
        - pyasn1 [required: >=0.4.6,<0.5.0, installed: 0.4.8]
      - rsa [required: >=3.1.4, installed: 4.8]
        - pyasn1 [required: >=0.1.3, installed: 0.4.8]
      - six [required: >=1.9.0, installed: 1.15.0]
    - google-auth-oauthlib [required: >=0.4.1,<0.5, installed: 0.4.6]
      - google-auth [required: >=1.0.0, installed: 1.6.3]
        - cachetools [required: >=2.0.0, installed: 5.0.0]
        - pyasn1-modules [required: >=0.2.1, installed: 0.2.8]
          - pyasn1 [required: >=0.4.6,<0.5.0, installed: 0.4.8]
        - rsa [required: >=3.1.4, installed: 4.8]
          - pyasn1 [required: >=0.1.3, installed: 0.4.8]
        - six [required: >=1.9.0, installed: 1.15.0]
      - requests-oauthlib [required: >=0.7.0, installed: 1.3.0]
        - oauthlib [required: >=3.0.0, installed: 3.1.1]
        - requests [required: >=2.0.0, installed: 2.27.1]
          - certifi [required: >=2017.4.17, installed: 2021.10.8]
          - charset-normalizer [required: ~=2.0.0, installed: 2.0.10]
          - idna [required: >=2.5,<4, installed: 3.3]
          - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.8]
    - grpcio [required: >=1.24.3, installed: 1.43.0]
      - six [required: >=1.5.2, installed: 1.15.0]
    - markdown [required: >=2.6.8, installed: 3.3.6]
      - importlib-metadata [required: >=4.4, installed: 4.10.0]
        - zipp [required: >=0.5, installed: 3.7.0]
    - numpy [required: >=1.12.0, installed: 1.22.1]
    - protobuf [required: >=3.6.0, installed: 3.19.3]
    - requests [required: >=2.21.0,<3, installed: 2.27.1]
      - certifi [required: >=2017.4.17, installed: 2021.10.8]
      - charset-normalizer [required: ~=2.0.0, installed: 2.0.10]
      - idna [required: >=2.5,<4, installed: 3.3]
      - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.8]
    - setuptools [required: >=41.0.0, installed: 60.5.0]
    - tensorboard-data-server [required: >=0.6.0,<0.7.0, installed: 0.6.1]
    - tensorboard-plugin-wit [required: >=1.6.0, installed: 1.8.1]
    - werkzeug [required: >=0.11.15, installed: 2.0.2]
    - wheel [required: >=0.26, installed: 0.37.1]
  - tensorflow-estimator [required: ~=2.7.0rc0,<2.8, installed: 2.7.0]
  - termcolor [required: >=1.1.0, installed: 1.1.0]
  - typing-extensions [required: >=3.6.6, installed: 4.0.1]
  - wheel [required: >=0.32.0,<1.0, installed: 0.37.1]
  - wrapt [required: >=1.11.0, installed: 1.13.3]

Note that tensorflow-io-gcs-filesystem is not listed and it is not installed.

I downloaded the wheel for tensorflow-macos and this is the content of the metadata file:

Metadata-Version: 2.1
Name: tensorflow-macos
Version: 2.7.0
Summary: TensorFlow is an open source machine learning framework for everyone.
Home-page: https://www.tensorflow.org/
Author: Google Inc.
Author-email: [e-mail]
License: Apache 2.0
Download-URL: https://github.com/tensorflow/tensorflow/tags
Keywords: tensorflow tensor machine learning
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: GPU :: NVIDIA CUDA :: 11.2
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Education
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Mathematics
Classifier: Topic :: Software Development
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy (>=1.14.5)
Requires-Dist: absl-py (>=0.4.0)
Requires-Dist: astunparse (>=1.6.0)
Requires-Dist: libclang (>=9.0.1)
Requires-Dist: flatbuffers (<3.0,>=1.12)
Requires-Dist: google-pasta (>=0.1.1)
Requires-Dist: h5py (>=2.9.0)
Requires-Dist: keras-preprocessing (>=1.1.1)
Requires-Dist: opt-einsum (>=2.3.2)
Requires-Dist: protobuf (>=3.9.2)
Requires-Dist: six (>=1.12.0)
Requires-Dist: termcolor (>=1.1.0)
Requires-Dist: typing-extensions (>=3.6.6)
Requires-Dist: wheel (<1.0,>=0.32.0)
Requires-Dist: wrapt (>=1.11.0)
Requires-Dist: gast (<0.5.0,>=0.2.1)
Requires-Dist: tensorboard (~=2.6)
Requires-Dist: tensorflow-estimator (<2.8,~=2.7.0rc0)
Requires-Dist: keras (<2.8,>=2.7.0rc0)
Requires-Dist: grpcio (<2.0,>=1.24.3)

[![Python](https://img.shields.io/pypi/pyversions/tensorflow.svg?style=plastic)](https://badge.fury.io/py/tensorflow)
[![PyPI](https://badge.fury.io/py/tensorflow.svg)](https://badge.fury.io/py/tensorflow)

TensorFlow is an open source software library for high performance numerical
computation. Its flexible architecture allows easy deployment of computation
across a variety of platforms (CPUs, GPUs, TPUs), and from desktops to clusters
of servers to mobile and edge devices.

Originally developed by researchers and engineers from the Google Brain team
within Google's AI organization, it comes with strong support for machine
learning and deep learning and the flexible numerical computation core is used
across many other scientific domains.

Note that tensorflow-io-gcs-filesystem is not listed as a dependency.

And here’s the setup.py dependency specification section:

REQUIRED_PACKAGES = [
    # NOTE: As numpy has releases that break semver guarantees and several other
    # deps depend on numpy without an upper bound, we must install numpy before
    # everything else.
    'numpy >= 1.14.5', # keras 2.6 needs 1.19.2, h5py needs 1.14.5 for py37
    # Install other dependencies
    'absl-py >= 0.4.0',
    'astunparse >= 1.6.0',
    'libclang >= 9.0.1',
    'flatbuffers >= 1.12, < 3.0', # capped as jax 0.1.71 needs < 3.0
    'google_pasta >= 0.1.1',
    'h5py >= 2.9.0', # capped since 3.3.0 lacks py3.6
    'keras_preprocessing >= 1.1.1', # 1.1.0 needs tensorflow==1.7
    'opt_einsum >= 2.3.2', # sphinx pin not removed up til 3.3.0 release
    'protobuf >= 3.9.2',
    'six >= 1.12.0',
    'termcolor >= 1.1.0',
    'typing_extensions >= 3.6.6',
    'wheel >= 0.32.0, < 1.0', # capped as astunparse 1.6.0-1.6.3 requires < 1.0
    'wrapt >= 1.11.0',
    # These packages need to be pinned exactly as newer versions are
    # incompatible with the rest of the ecosystem
    'gast >= 0.2.1, < 0.5.0', # TODO(lpak): if this breaks, revert to 0.4.0
    # TensorFlow ecosystem packages that TF exposes API for
    # These need to be in sync with the existing TF version
    # They are updated during the release process
    # When updating these, please also update the nightly versions below
    'tensorboard ~= 2.6',
    'tensorflow_estimator ~= 2.7.0rc0, < 2.8',
    # Keras release is not backward compatible with old tf release, and we have
    # to make the version aligned between TF and Keras.
    'keras >= 2.7.0rc0, < 2.8',
    # 'tensorflow-io-gcs-filesystem >= 0.21.0',
]

Note that the tensorflow-io-gcs-filesystem is listed but has been commented out.

So, my question is how is poetry coming to a conclusion that tensorflow-io-gcs-filesystem is a dependency of tensorflow-macos?

I am asking because I am unable to install tensorflow-macos using poetry because of this reason. poetry install shows the following error:

  • Installing tensorflow-io-gcs-filesystem (0.23.1): Failed

  RuntimeError

  Unable to find installation candidates for tensorflow-io-gcs-filesystem (0.23.1)

  at ~/.poetry/lib/poetry/installation/chooser.py:72 in choose_for
       68│
       69│             links.append(link)
       70│
       71│         if not links:
    →  72│             raise RuntimeError(
       73│                 "Unable to find installation candidates for {}".format(package)
       74│             )
       75│
       76│         # Get the best link

This is understandable because tensorflow-io-gcs-filesystem does not have binaries for macOS arm64. But why is it considered a dependency in the first place?

[oscar-defelice]

I have a very similar problem and this is not due to python version nor to tensorflow dependency.

I run poetry add tensorflow==2.9.1

 Installing tensorflow-io-gcs-filesystem (0.26.0): Failed

  RuntimeError

  Unable to find installation candidates for tensorflow-io-gcs-filesystem (0.26.0)

  at ~/miniconda3/envs/tempr/lib/python3.9/site-packages/poetry/installation/chooser.py:72 in choose_for
       68│ 
       69│             links.append(link)
       70│ 
       71│         if not links:
    →  72│             raise RuntimeError(
       73│                 "Unable to find installation candidates for {}".format(package)
       74│             )
       75│ 
       76│         # Get the best link

[radoering]

@unredundant The python constraint in your pyproject.toml is too loose. See Why is Poetry telling me that the current project’s Python requirement is not compatible with one or more packages’ Python requirements?.