Can't resolve dependencies for Gensim / smart-open / botocore

Issue description

pipenv install gensim fails to create Pipfile.lock because of incompatible required versions in smart-open of botocore.

Expected result

It should install without conflict … somehow

Actual result

pipenv graph gives this:

gensim==3.5.0
  - numpy [required: >=1.11.3, installed: 1.15.1]
  - scipy [required: >=0.18.1, installed: 1.1.0]
    - numpy [required: >=1.8.2, installed: 1.15.1]
  - six [required: >=1.5.0, installed: 1.11.0]
  - smart-open [required: >=1.2.1, installed: 1.7.0]
    - boto [required: >=2.32, installed: 2.49.0]
    - boto3 [required: <1.8.0, installed: 1.7.84]
      - botocore [required: >=1.10.84,<1.11.0, installed: 1.10.84]
        - docutils [required: >=0.10, installed: 0.14]
        - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.3]
        - python-dateutil [required: >=2.1,<3.0.0, installed: 2.7.3]
          - six [required: >=1.5, installed: 1.11.0]
      - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.3]
      - s3transfer [required: >=0.1.10,<0.2.0, installed: 0.1.13]
        - botocore [required: >=1.3.0,<2.0.0, installed: 1.10.84]
          - docutils [required: >=0.10, installed: 0.14]
          - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.3]
          - python-dateutil [required: >=2.1,<3.0.0, installed: 2.7.3]
            - six [required: >=1.5, installed: 1.11.0]
    - botocore [required: <1.11.0, installed: 1.10.84]
      - docutils [required: >=0.10, installed: 0.14]
      - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.3]
      - python-dateutil [required: >=2.1,<3.0.0, installed: 2.7.3]
        - six [required: >=1.5, installed: 1.11.0]
    - bz2file [required: Any, installed: 0.98]
    - requests [required: Any, installed: 2.19.1]
      - certifi [required: >=2017.4.17, installed: 2018.8.24]
      - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
      - idna [required: >=2.5,<2.8, installed: 2.7]
      - urllib3 [required: >=1.21.1,<1.24, installed: 1.23]

Specifically, please note these three lines inside smart-open:

- botocore [required: >=1.10.84,<1.11.0,  installed: 1.10.84]
- botocore [required: >=1.3.0,<2.0.0,     installed: 1.10.84]
- botocore [required: <1.11.0,            installed: 1.10.84]

This may be a problem with smart-open rather than pipenv but I can’t tell. I posted the issue on smart-open but I don’t know enough to decide who needs to adjust what: https://github.com/RaRe-Technologies/smart_open/issues/227

Steps to replicate

pipenv install gensim Provide the steps to replicate (which usually at least includes the commands and the Pipfile).

pipenv --support:

$ pipenv --support

Pipenv version: '2018.7.1'

Pipenv location: '/usr/local/Cellar/pipenv/2018.7.1/libexec/lib/python3.7/site-packages/pipenv'

Python location: '/usr/local/Cellar/pipenv/2018.7.1/libexec/bin/python3.7'

Other Python installations in PATH:

• 2.7: /usr/local/bin/python2.7

• 2.7: /usr/local/bin/python2.7

• 2.7: /usr/bin/python2.7

• 3.6: /Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6m

• 3.6: /Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6

• 3.6: /usr/local/bin/python3.6

• 3.7: /Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7m

• 3.7: /Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7

• 3.7: /usr/local/bin/python3.7

• 2.7.15: /usr/local/bin/python

• 2.7.10: /usr/bin/python

• 2.7.15: /usr/local/bin/python2

• 3.7.0: /Library/Frameworks/Python.framework/Versions/3.7/bin/python3

• 3.6.4: /Library/Frameworks/Python.framework/Versions/3.6/bin/python3

• 3.7.0: /usr/local/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.0',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '17.7.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT '
                     '2018; root:xnu-4570.71.2~1/RELEASE_X86_64',
 'python_full_version': '3.7.0',
 'python_version': '3.7',
 'sys_platform': 'darwin'}

System environment variables:

• PATH
• TERM_PROGRAM
• NVM_CD_FLAGS
• TERM
• SHELL
• TMPDIR
• Apple_PubSub_Socket_Render
• TERM_PROGRAM_VERSION
• ZSH
• NVM_DIR
• USER
• SSH_AUTH_SOCK
• __CF_USER_TEXT_ENCODING
• PAGER
• LSCOLORS
• _
• PWD
• LANG
• XPC_FLAGS
• XPC_SERVICE_NAME
• SHLVL
• HOME
• LESS
• LOGNAME
• LC_CTYPE
• NVM_BIN
• DISPLAY
• COLORTERM
• PYTHONDONTWRITEBYTECODE
• PIP_PYTHON_PATH

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /usr/local/Cellar/pipenv/2018.7.1/libexec/tools:/usr/local/opt/opencv3/bin:/Users/byakubchik/.nvm/versions/node/v8.11.2/bin:/Library/Frameworks/Python.framework/Versions/3.7/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Library/Frameworks/Python.framework/Versions/2.7/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:~/.composer/vendor/bin:/Library/TeX/texbin:/usr/local/go/bin:/opt/X11/bin
• SHELL: /bin/zsh
• LANG: en_US.UTF-8
• PWD: /Users/byakubchik/Desktop/temp

---

Contents of Pipfile (‘/Users/byakubchik/Desktop/temp/Pipfile’):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
gensim = "*"

[dev-packages]

[requires]
python_version = "3.7"

[whyboris]

The issue was fixed very swiftly at smart_open with a new release that unpins the required dependency.

https://github.com/RaRe-Technologies/smart_open/issues/227#issuecomment-422859000

I believe the issue I raised is an instance of an inevitable problem when package A wants sub-dependency C at version 1.0 while package B wants sub-dependency C at version 2.0 – I’m rather surprised you didn’t run into a problem.

@petergaultney - I was unable to replicate the error now that the package has been updated, but I too am curious why you experienced different behavior between the two versions of Python.

[uranusjr]

Yes, the cache is partitioned by version.