poetry: S3 feature is incompatible with latest boto3 version

Bug Report

Issue names must follow the pattern command: description where the command is the dvc command that you are trying to run. The description should describe the consequence of the bug.

This time the bug isn’t in a DVC command but in how its dependencies are declared. Feel free to rename the issue to match the preferred format.

Description

I’ve installed the latest version of the boto3 package for my application.

I want to use it together with DVC’s S3 feature.

When I try to install the S3 feature, I get a SolverProblemError from poetry.

Reproduce

$ mkdir tmp
$ cd tmp
$ poetry init

This command will guide you through creating your pyproject.toml config.

Package name [tmp]:  
Version [0.1.0]:  
Description []:  
Author [Iain Samuel McLean Elder <[e-mail]>, n to skip]:  
License []:  
Compatible Python versions [^3.8]:  

Would you like to define your main dependencies interactively? (yes/no) [yes] no
Would you like to define your development dependencies interactively? (yes/no) [yes] no
Generated file

[tool.poetry]
name = "tmp"
version = "0.1.0"
description = ""
authors = ["Iain Samuel McLean Elder <[e-mail]>"]

[tool.poetry.dependencies]
python = "^3.8"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"


Do you confirm generation? (yes/no) [yes] 
$ poetry add boto3
Creating virtualenv tmp-pXV3aO_m-py3.8 in /home/isme/.cache/pypoetry/virtualenvs
Using version ^1.23.1 for boto3

Updating dependencies
Resolving dependencies... (0.5s)

Writing lock file

Package operations: 7 installs, 0 updates, 0 removals

  • Installing six (1.16.0)
  • Installing jmespath (1.0.0)
  • Installing python-dateutil (2.8.2)
  • Installing urllib3 (1.26.9)
  • Installing botocore (1.26.1)
  • Installing s3transfer (0.5.2)
  • Installing boto3 (1.23.1)
$ poetry add dvc[s3]
Using version ^2.10.2 for dvc

Updating dependencies
Resolving dependencies... (9.8s)

  SolverProblemError

      Because aiobotocore (2.0.1) depends on botocore (>=1.22.8,<1.22.9)
   and no versions of aiobotocore match >2.0.1,<2.1.0, aiobotocore (>=2.0.1,<2.1.0) requires botocore (>=1.22.8,<1.22.9).
      And because s3fs (2021.11.1) depends on aiobotocore (>=2.0.1,<2.1.0)
   and no versions of s3fs match >2021.11.1,<2022.1.0 || >2022.1.0,<2022.2.0 || >2022.2.0,<2022.3.0 || >2022.3.0, s3fs (>=2021.11.1,<2022.1.0 || >2022.1.0,<2022.2.0 || >2022.2.0,<2022.3.0 || >2022.3.0) requires botocore (>=1.22.8,<1.22.9).
  (1) So, because s3fs (2022.2.0) depends on aiobotocore (>=2.1.0,<2.2.0)
   and s3fs (2022.1.0) depends on aiobotocore (>=2.1.0,<2.2.0), s3fs (>=2021.11.1,<2022.3.0 || >2022.3.0) requires botocore (>=1.22.8,<1.22.9) or aiobotocore (>=2.1.0,<2.2.0).
  
      Because no versions of aiobotocore match >2.1.0,<2.1.1 || >2.1.1,<2.1.2 || >2.1.2,<2.2.0
   and aiobotocore (2.1.0) depends on botocore (>=1.23.24,<1.23.25), aiobotocore (>=2.1.0,<2.1.1 || >2.1.1,<2.1.2 || >2.1.2,<2.2.0) requires botocore (>=1.23.24,<1.23.25).
      And because aiobotocore (2.1.1) depends on botocore (>=1.23.24,<1.23.25)
   and aiobotocore (2.1.2) depends on botocore (>=1.23.24,<1.23.25), aiobotocore (>=2.1.0,<2.2.0) requires botocore (>=1.23.24,<1.23.25).
      And because s3fs (>=2021.11.1,<2022.3.0 || >2022.3.0) requires botocore (>=1.22.8,<1.22.9) or aiobotocore (>=2.1.0,<2.2.0) (1), s3fs (>=2021.11.1,<2022.3.0 || >2022.3.0) requires botocore (>=1.22.8,<1.22.9 || >=1.23.24,<1.23.25)
      And because s3fs (2022.3.0) depends on aiobotocore (>=2.2.0,<2.3.0), s3fs (>=2021.11.1) requires botocore (>=1.22.8,<1.22.9 || >=1.23.24,<1.23.25) or aiobotocore (>=2.2.0,<2.3.0).
      Because no versions of aiobotocore match >2.2.0,<2.3.0
   and aiobotocore (2.2.0) depends on botocore (>=1.24.21,<1.24.22), aiobotocore (>=2.2.0,<2.3.0) requires botocore (>=1.24.21,<1.24.22).
      Thus, s3fs (>=2021.11.1) requires botocore (>=1.22.8,<1.22.9 || >=1.23.24,<1.23.25 || >=1.24.21,<1.24.22).
      Because no versions of dvc match >2.10.2,<3.0.0
   and dvc (2.10.2) depends on s3fs (>=2021.11.1), dvc (>=2.10.2,<3.0.0) requires s3fs (>=2021.11.1).
      Thus, dvc (>=2.10.2,<3.0.0) requires botocore (>=1.22.8,<1.22.9 || >=1.23.24,<1.23.25 || >=1.24.21,<1.24.22).
      And because boto3 (1.23.1) depends on botocore (>=1.26.1,<1.27.0)
   and no versions of boto3 match >1.23.1,<2.0.0, dvc (>=2.10.2,<3.0.0) is incompatible with boto3 (>=1.23.1,<2.0.0).
      So, because tmp depends on both boto3 (^1.23.1) and dvc (^2.10.2), version solving failed.

  at ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│ 
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes

Expected

No error message. I expect that the S3 feature will be installed with whatever version version of boto3 I’m using in my application.

Environment information

$ poetry run dvc doctor
DVC version: 2.10.2 (pip)
---------------------------------
Platform: Python 3.8.10 on Linux-5.13.0-41-generic-x86_64-with-glibc2.29
Supports:
	webhdfs (fsspec = 2022.3.0),
	http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
	https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/nvme0n1p3
Caches: local
Remotes: s3
Workspace directory: ext4 on /dev/nvme0n1p3
Repo: dvc, git

[daavoo]

Unfortunately, it seems that until aiobotocore updates the pinned boto3 requirement (https://github.com/aio-libs/aiobotocore/blob/master/setup.py#L19) , the most recent compatible version would be 1.21.21:

poetry add boto3==1.21.21
poetry add 'dvc[s3]'

[daavoo]

Can also reproduce with pip resolver:

pip install boto3==1.23.1 'dvc[s3]==2.10.2'

ERROR: Cannot install aiobotocore[boto3]==2.0.1, aiobotocore[boto3]==2.1.0, aiobotocore[boto3]==2.1.1, aiobotocore[boto3]==2.1.2, aiobotocore[boto3]==2.2.0, aiobotocore[boto3]==2.3.0, aiobotocore[boto3]==2.3.1, aiobotocore[boto3]==2.3.2 and boto3==1.23.1 because these package versions have conflicting dependencies.

The conflict is caused by:
    boto3 1.23.1 depends on botocore<1.27.0 and >=1.26.1
    aiobotocore[boto3] 2.3.2 depends on botocore<1.24.22 and >=1.24.21
    boto3 1.23.1 depends on botocore<1.27.0 and >=1.26.1
    aiobotocore[boto3] 2.3.1 depends on botocore<1.24.22 and >=1.24.21
    boto3 1.23.1 depends on botocore<1.27.0 and >=1.26.1
    aiobotocore[boto3] 2.3.0 depends on botocore<1.24.22 and >=1.24.21
    boto3 1.23.1 depends on botocore<1.27.0 and >=1.26.1
    aiobotocore[boto3] 2.2.0 depends on botocore<1.24.22 and >=1.24.21
    boto3 1.23.1 depends on botocore<1.27.0 and >=1.26.1
    aiobotocore[boto3] 2.1.2 depends on botocore<1.23.25 and >=1.23.24
    boto3 1.23.1 depends on botocore<1.27.0 and >=1.26.1
    aiobotocore[boto3] 2.1.1 depends on botocore<1.23.25 and >=1.23.24
    boto3 1.23.1 depends on botocore<1.27.0 and >=1.26.1
    aiobotocore[boto3] 2.1.0 depends on botocore<1.23.25 and >=1.23.24
    boto3 1.23.1 depends on botocore<1.27.0 and >=1.26.1
    aiobotocore[boto3] 2.0.1 depends on botocore<1.22.9 and >=1.22.8

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict