Pip resolver can't install elasticsearch-curator due to incompatible urllib3 version pins

What did you want to do?

Install elasticsearch-curator package with pip v20.3.dev1:

$ python -m pip install elasticsearch-curator

Output

Collecting elasticsearch-curator
  Using cached elasticsearch-curator-5.8.1.tar.gz (225 kB)
    ERROR: Command errored out with exit status 1:
     command: /tmp/venv/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/setup.py'"'"'; __file__='"'"'/tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-9npyg8qm
         cwd: /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/
    Complete output (135 lines):
    build/temp.linux-x86_64-3.6/check_libyaml.c:2:10: fatal error: yaml.h: No such file or directory
     #include <yaml.h>
              ^~~~~~~~
    compilation terminated.
    
    libyaml is not found or a compiler error: forcing --without-libyaml
    (if libyaml is installed correctly, you may need to
     specify the option --include-dirs or uncomment and
     modify the parameter include_dirs in setup.cfg)
    zip_safe flag not set; analyzing archive contents...
    
    Installed /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs/PyYAML-3.13-py3.6-linux-x86_64.egg
    Searching for click<7.0,>=6.7
    Reading https://pypi.python.org/simple/click/
    Downloading https://files.pythonhosted.org/packages/34/c1/8806f99713ddb993c5366c362b2f908f18269f8d792aff1abfd700775a77/click-6.7-py2.py3-none-any.whl#sha256=29f99fc6125fbc931b758dc053b3114e55c77a6e4c6c3a2674a2dc986016381d
    Best match: click 6.7
    Processing click-6.7-py2.py3-none-any.whl
    Installing click-6.7-py2.py3-none-any.whl to /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs
    
    Installed /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs/click-6.7-py3.6.egg
    Searching for requests_aws4auth>=0.9
    Reading https://pypi.python.org/simple/requests_aws4auth/
    Downloading https://files.pythonhosted.org/packages/4e/44/dec6791ef89cef395918f6e4268d0f30096ee798523fcf26edd53117210a/requests_aws4auth-1.0.1-py2.py3-none-any.whl#sha256=6ddc92476c0ebab6c90709966e0ff3b2f16ba1f296b7ea8573645bbbe5ef9444
    Best match: requests-aws4auth 1.0.1
    Processing requests_aws4auth-1.0.1-py2.py3-none-any.whl
    Installing requests_aws4auth-1.0.1-py2.py3-none-any.whl to /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs
    writing requirements to /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs/requests_aws4auth-1.0.1-py3.6.egg/EGG-INFO/requires.txt
    
    Installed /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs/requests_aws4auth-1.0.1-py3.6.egg
    Searching for boto3>=1.9.142
    Reading https://pypi.python.org/simple/boto3/
    Downloading https://files.pythonhosted.org/packages/60/06/78b1aa1768cfbab8a333db53101b69b800cd357d4dec7625de872ba5b453/boto3-1.16.12-py2.py3-none-any.whl#sha256=74c6c1757178b08cf7d43238473b8557342f007491d435b2beb312d0fe67c62f
    Best match: boto3 1.16.12
    Processing boto3-1.16.12-py2.py3-none-any.whl
    Installing boto3-1.16.12-py2.py3-none-any.whl to /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs
    writing requirements to /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs/boto3-1.16.12-py3.6.egg/EGG-INFO/requires.txt
    
    Installed /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs/boto3-1.16.12-py3.6.egg
    Searching for requests>=2.20.0
    Reading https://pypi.python.org/simple/requests/
    Downloading https://files.pythonhosted.org/packages/45/1e/0c169c6a5381e241ba7404532c16a21d86ab872c9bed8bdcd4c423954103/requests-2.24.0-py2.py3-none-any.whl#sha256=fe75cc94a9443b9246fc7049224f75604b113c36acb93f87b80ed42c44cbb898
    Best match: requests 2.24.0
    Processing requests-2.24.0-py2.py3-none-any.whl
    Installing requests-2.24.0-py2.py3-none-any.whl to /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs
    writing requirements to /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs/requests-2.24.0-py3.6.egg/EGG-INFO/requires.txt
    
    Installed /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs/requests-2.24.0-py3.6.egg
    Searching for urllib3<1.25,>=1.24.2
    Reading https://pypi.python.org/simple/urllib3/
    Downloading https://files.pythonhosted.org/packages/01/11/525b02e4acc0c747de8b6ccdab376331597c569c42ea66ab0a1dbd36eca2/urllib3-1.24.3-py2.py3-none-any.whl#sha256=a637e5fae88995b256e3409dc4d52c2e2e0ba32c42a6365fee8bbd2238de3cfb
    Best match: urllib3 1.24.3
    Processing urllib3-1.24.3-py2.py3-none-any.whl
    Installing urllib3-1.24.3-py2.py3-none-any.whl to /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs
    writing requirements to /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs/urllib3-1.24.3-py3.6.egg/EGG-INFO/requires.txt
    
    Installed /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs/urllib3-1.24.3-py3.6.egg
    Searching for elasticsearch<8.0.0,>=7.0.4
    Reading https://pypi.python.org/simple/elasticsearch/
    Downloading https://files.pythonhosted.org/packages/38/5c/c3efebe8065bd4f9e815751009841934b8fcd7bfba93362a54384a04fd8b/elasticsearch-7.10.0a2-py2.py3-none-any.whl#sha256=4883b42428fc073b2bb19c2c4d13ad4c951e2f1bde73c40dc2f2ae7bbd3c3526
    Best match: elasticsearch 7.10.0a2
    Processing elasticsearch-7.10.0a2-py2.py3-none-any.whl
    Installing elasticsearch-7.10.0a2-py2.py3-none-any.whl to /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs
    writing requirements to /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs/elasticsearch-7.10.0a2-py3.6.egg/EGG-INFO/requires.txt
    
    Installed /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs/elasticsearch-7.10.0a2-py3.6.egg
    Searching for voluptuous>=0.9.3
    Reading https://pypi.python.org/simple/voluptuous/
    Downloading https://files.pythonhosted.org/packages/d3/e9/861cec3f4da0728d9d085db9c338b598351cbacb10521c25f0857f55c6d4/voluptuous-0.12.0-py3-none-any.whl#sha256=0fff348a097c9a74f9f4a991d2cf01a6185780e997ad953bde49cb3efbb411be
    Best match: voluptuous 0.12.0
    Processing voluptuous-0.12.0-py3-none-any.whl
    Installing voluptuous-0.12.0-py3-none-any.whl to /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs
    
    Installed /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs/voluptuous-0.12.0-py3.6.egg
    Searching for s3transfer<0.4.0,>=0.3.0
    Reading https://pypi.python.org/simple/s3transfer/
    Downloading https://files.pythonhosted.org/packages/69/79/e6afb3d8b0b4e96cefbdc690f741d7dd24547ff1f94240c997a26fa908d3/s3transfer-0.3.3-py2.py3-none-any.whl#sha256=2482b4259524933a022d59da830f51bd746db62f047d6eb213f2f8855dcb8a13
    Best match: s3transfer 0.3.3
    Processing s3transfer-0.3.3-py2.py3-none-any.whl
    Installing s3transfer-0.3.3-py2.py3-none-any.whl to /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs
    writing requirements to /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs/s3transfer-0.3.3-py3.6.egg/EGG-INFO/requires.txt
    
    Installed /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs/s3transfer-0.3.3-py3.6.egg
    Searching for jmespath<1.0.0,>=0.7.1
    Reading https://pypi.python.org/simple/jmespath/
    Downloading https://files.pythonhosted.org/packages/07/cb/5f001272b6faeb23c1c9e0acc04d48eaaf5c862c17709d20e3469c6e0139/jmespath-0.10.0-py2.py3-none-any.whl#sha256=cdf6525904cc597730141d61b36f2e4b8ecc257c420fa2f4549bac2c2d0cb72f
    Best match: jmespath 0.10.0
    Processing jmespath-0.10.0-py2.py3-none-any.whl
    Installing jmespath-0.10.0-py2.py3-none-any.whl to /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs
    
    Installed /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs/jmespath-0.10.0-py3.6.egg
    Searching for botocore<1.20.0,>=1.19.12
    Reading https://pypi.python.org/simple/botocore/
    Downloading https://files.pythonhosted.org/packages/24/7f/9298904c3b9b6f6e49f1cda7e3c92c8b2e8639020e11091fdfa2a345b354/botocore-1.19.12-py2.py3-none-any.whl#sha256=d0a39209551381752ca5257ae9b3eef53e7fe7eb16ccb9854fb1b14bc3d43519
    Best match: botocore 1.19.12
    Processing botocore-1.19.12-py2.py3-none-any.whl
    Installing botocore-1.19.12-py2.py3-none-any.whl to /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs
    writing requirements to /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs/botocore-1.19.12-py3.6.egg/EGG-INFO/requires.txt
    
    Installed /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs/botocore-1.19.12-py3.6.egg
    Searching for idna<3,>=2.5
    Reading https://pypi.python.org/simple/idna/
    Downloading https://files.pythonhosted.org/packages/a2/38/928ddce2273eaa564f6f50de919327bf3a00f091b5baba8dfa9460f3a8a8/idna-2.10-py2.py3-none-any.whl#sha256=b97d804b1e9b523befed77c48dacec60e6dcb0b5391d57af6a65a312a90648c0
    Best match: idna 2.10
    Processing idna-2.10-py2.py3-none-any.whl
    Installing idna-2.10-py2.py3-none-any.whl to /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs
    
    Installed /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs/idna-2.10-py3.6.egg
    Searching for chardet<4,>=3.0.2
    Reading https://pypi.python.org/simple/chardet/
    Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl#sha256=fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691
    Best match: chardet 3.0.4
    Processing chardet-3.0.4-py2.py3-none-any.whl
    Installing chardet-3.0.4-py2.py3-none-any.whl to /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs
    
    Installed /tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs/chardet-3.0.4-py3.6.egg
    Traceback (most recent call last):
      File "/tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/setup.py", line 39, in <module>
        from cx_Freeze import setup, Executable
    ModuleNotFoundError: No module named 'cx_Freeze'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/setup.py", line 177, in <module>
        tests_require = ["mock", "nose", "coverage", "nosexcover"]
      File "/tmp/venv/lib/python3.6/site-packages/setuptools/__init__.py", line 128, in setup
        _install_setup_requires(attrs)
      File "/tmp/venv/lib/python3.6/site-packages/setuptools/__init__.py", line 123, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/tmp/venv/lib/python3.6/site-packages/setuptools/dist.py", line 513, in fetch_build_eggs
        replace_conflicting=True,
      File "/tmp/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 783, in resolve
        raise VersionConflict(dist, req).with_context(dependent_req)
    pkg_resources.ContextualVersionConflict: (urllib3 1.24.3 (/tmp/pip-install-dndpdbmp/elasticsearch-curator_7f80ee0153e84d61a6cf1c16b82db8a4/.eggs/urllib3-1.24.3-py3.6.egg), Requirement.parse('urllib3<1.26,>=1.25.4'), {'botocore'})
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

The elasticsearch-curator package has a dependency on boto3 and urllib3<1.25, boto3 has dependency on recent botocore, recent botocore versions have a dependency on urllib3>=1.25.9. This situation isn’t resolved by pip.

Curator issue: https://github.com/elastic/curator/issues/1560

[pradyunsg]

https://github.com/elastic/curator/blob/99bb6d46823ac33fed399f5b0786e33dee2e9058/setup.py#L115

The issue seems to be the use of setup_requires, which triggers easy_install for those versions. easy_install will 100% blow up in your face like it did here. 😃

I’m not sure why you need setup_requires here, but that’s almost certainly replaced by specifying the build-system table in a pyproject.toml.

[sethmlarson]

@uranusjr From the output I provided pip selected the following incompatible dists and installed them in this order:

• boto3==1.16.12 (botocore>=1.19,<1.20)
• urllib3==1.24.3
• botocore==1.19.12 (urllib3>=1.25.9)
• elasticsearch-curator==5.8.1 (urllib3<1.25, boto3)

As far as I know it’s not setuptools making the decision which dists and versions are installed. Setuptools only raises the exception when the version conflict is discovered when trying to finally install elasticsearch-curator. Can you confirm you’re also seeing this / my understanding of the situation is correct?