Public dependencies are not resolved properly on private packages

When a public package is listed as a dependency in a private package, the package will fail to install because it doesn’t exist in the private index.

For example, if you specify an additional source in your Pipfile:

[[source]]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"


[[source]]

url = "http://localhost:8080/"
name = "local"

This is the following output from running pipenv install --verbose local_package

Installing local_package…
⠋Installing u'local_package'
$ "/Users/guido/.local/share/virtualenvs/src-GqcYiTnw/bin/pip" install   --verbose    "local_package" -i https://pypi.python.org/simple --exists-action w
⠏$ "/Users/guido/.local/share/virtualenvs/src-GqcYiTnw/bin/pip" install   --verbose    "local_package" -i http://localhost:8080/ --exists-action w
Collecting local_package
  1 location(s) to search for versions of local-package:
  * http://localhost:8080/local-package/
  Getting page http://localhost:8080/local-package/
  Starting new HTTP connection (1): localhost
  "GET /local-package/ HTTP/1.1" 200 None
  Analyzing links from page http://localhost:8080/local-package/
    Found link http://localhost:8080/local-package/local_package-0.1.1-py3-none-any.whl#md5=ce978ae83f1fec0526e7f34c45d27810 (from http://localhost:8080/local-package/), version: 0.1.1
  Using version 0.1.1 (newest of versions: 0.1.1)
  "GET /local-package/local_package-0.1.1-py3-none-any.whl HTTP/1.1" 200 7273
  Downloading http://localhost:8080/local-package/local_package-0.1.1-py3-none-any.whl
  Downloading from URL http://localhost:8080/local-package/local_package-0.1.1-py3-none-any.whl#md5=ce978ae83f1fec0526e7f34c45d27810 (from http://localhost:8080/local-package/)
Collecting boto3 (from local_package)
  1 location(s) to search for versions of boto3:
  * http://localhost:8080/boto3/
  Getting page http://localhost:8080/boto3/
  "GET /boto3/ HTTP/1.1" 403 None
  Could not fetch URL http://localhost:8080/boto3/: 403 Client Error: Forbidden for url: http://localhost:8080/boto3/ - skipping
Cleaning up...
Exception information:
Traceback (most recent call last):
  File "/Users/guido/.local/share/virtualenvs/src-GqcYiTnw/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/Users/guido/.local/share/virtualenvs/src-GqcYiTnw/lib/python3.6/site-packages/pip/commands/install.py", line 335, in run
    wb.build(autobuilding=True)
  File "/Users/guido/.local/share/virtualenvs/src-GqcYiTnw/lib/python3.6/site-packages/pip/wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "/Users/guido/.local/share/virtualenvs/src-GqcYiTnw/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/Users/guido/.local/share/virtualenvs/src-GqcYiTnw/lib/python3.6/site-packages/pip/req/req_set.py", line 554, in _prepare_file
    require_hashes
  File "/Users/guido/.local/share/virtualenvs/src-GqcYiTnw/lib/python3.6/site-packages/pip/req/req_install.py", line 278, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/Users/guido/.local/share/virtualenvs/src-GqcYiTnw/lib/python3.6/site-packages/pip/index.py", line 514, in find_requirement
    'No matching distribution found for %s' % req
pip.exceptions.DistributionNotFound: No matching distribution found for boto3 (from local_package)

Error:  An error occurred while installing local_package!
  Could not find a version that satisfies the requirement boto3 (from local_package) (from versions: )
No matching distribution found for boto3 (from local_package)

It seems that the issue is with pipenv using -i instead of --extra-index-url

If I run pip install --extra-index-url http://localhost:8080 local_package everything is installed correctly.

Specifying the index for local_package under [packages] doesn’t help.

Describe your environment

1. OS Type: macOS Sierra
2. Python version: Python 3.6.3
3. Pipenv version: pipenv, version 10.1.2

Expected result

I would expect for the public package (in the case above, boto3), to be fetched from the public source by default.

Actual result

The public package is fetched from the private source.

Steps to replicate

Set up a private pypi index, add source to Pipfile, and run install command from above.

[ryanwilsonperkin]

Hey @kennethreitz, I don’t think d5b9b36 has quite solved this issue. I’ve attempted to reproduce by publishing a package to Test PyPI that references a package on regular PyPI.

---

Expected behaviour

Installs pipenv-test-private-package from TestPyPI and discovers dependency on pipenv-test-public-package which is available on regular PyPI.

Works when running pip directly:

/private/tmp/pip  (venv) pip install --extra-index-url https://testpypi.python.org/pypi pipenv-test-private-package
Collecting pipenv-test-private-package
  Downloading pipenv-test-private-package-1.0.tar.gz
Collecting pipenv-test-public-package (from pipenv-test-private-package)
  Downloading pipenv-test-public-package-1.0.tar.gz
Building wheels for collected packages: pipenv-test-private-package, pipenv-test-public-package
  Running setup.py bdist_wheel for pipenv-test-private-package ... done
  Stored in directory: /Users/ryan/Library/Caches/pip/wheels/2e/79/07/9f90adc981af1f8dd8fb1478c14f40228d0820b619d6e6b507
  Running setup.py bdist_wheel for pipenv-test-public-package ... done
  Stored in directory: /Users/ryan/Library/Caches/pip/wheels/21/0f/7d/581265ed90f2c7539e6ea64e5eb118ffbe652a4d6bade55d44
Successfully built pipenv-test-private-package pipenv-test-public-package
Installing collected packages: pipenv-test-public-package, pipenv-test-private-package
Successfully installed pipenv-test-private-package-1.0 pipenv-test-public-package-1.0

Actual behaviour

Fails with the following error message

/p/t/pipenv pipenv install --verbose
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Using pip: -i https://pypi.python.org/simple --extra-index-url https://testpypi.python.org/pypi

                          ROUND 1
Current constraints:

Finding the best candidates:

Finding secondary dependencies:
------------------------------------------------------------
Result of round 1: stable, done

Locking [packages] dependencies…
Using pip: -i https://pypi.python.org/simple --extra-index-url https://testpypi.python.org/pypi

                          ROUND 1
Current constraints:
  pipenv-test-private-package

Finding the best candidates:
  found candidate pipenv-test-private-package==1.0 (constraint was <any>)

Finding secondary dependencies:
  pipenv-test-private-package==1.0 not in cache, need to check index
  pipenv-test-private-package==1.0 requires pipenv-test-public-package

New dependencies found in this round:
  adding [u'pipenv-test-public-package', '', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2
Current constraints:
  pipenv-test-private-package
  pipenv-test-public-package

Finding the best candidates:
  found candidate pipenv-test-private-package==1.0 (constraint was <any>)

INFO:pip._vendor.requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): testpypi.python.org
INFO:pip._vendor.requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org
INFO:pip.download:Using cached pipenv-test-private-package-1.0.tar.gz
INFO:pip.download:Saved /Users/ryan/Library/Caches/pipenv/pkgs/pipenv-test-private-package-1.0.tar.gz
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/pipenv/resolver.py", line 66, in <module>
    main()
  File "/usr/local/lib/python2.7/site-packages/pipenv/resolver.py", line 54, in main
    results = resolve(packages, pre=do_pre, sources=project.sources, verbose=is_verbose, clear=do_clear)
  File "/usr/local/lib/python2.7/site-packages/pipenv/resolver.py", line 52, in resolve
    return pipenv.utils.resolve_deps(packages, which, project=project, pre=pre, sources=sources, clear=clear, verbose=verbose)
  File "/usr/local/lib/python2.7/site-packages/pipenv/../pipenv/utils.py", line 398, in resolve_deps
    resolved_tree, resolver = actually_resolve_reps(deps, index_lookup, markers_lookup, project, sources, verbose, clear, pre)
  File "/usr/local/lib/python2.7/site-packages/pipenv/../pipenv/utils.py", line 328, in actually_resolve_reps
    resolved_tree.update(resolver.resolve(max_rounds=PIPENV_MAX_ROUNDS))
  File "/usr/local/lib/python2.7/site-packages/pipenv/../pipenv/patched/piptools/resolver.py", line 102, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/usr/local/lib/python2.7/site-packages/pipenv/../pipenv/patched/piptools/resolver.py", line 193, in _resolve_one_round
    best_matches = {self.get_best_match(ireq) for ireq in constraints}
  File "/usr/local/lib/python2.7/site-packages/pipenv/../pipenv/patched/piptools/resolver.py", line 193, in <setcomp>
    best_matches = {self.get_best_match(ireq) for ireq in constraints}
  File "/usr/local/lib/python2.7/site-packages/pipenv/../pipenv/patched/piptools/resolver.py", line 257, in get_best_match
    best_match = self.repository.find_best_match(ireq, prereleases=self.prereleases)
  File "/usr/local/lib/python2.7/site-packages/pipenv/../pipenv/patched/piptools/repositories/pypi.py", line 114, in find_best_match
    all_candidates = self.find_all_candidates(ireq.name)
  File "/usr/local/lib/python2.7/site-packages/pipenv/../pipenv/patched/piptools/repositories/pypi.py", line 99, in find_all_candidates
    candidates = self.finder.find_all_candidates(req_name)
  File "/usr/local/lib/python2.7/site-packages/pipenv/patched/notpip/index.py", line 456, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/usr/local/lib/python2.7/site-packages/pipenv/patched/notpip/index.py", line 605, in _get_pages
    page = self._get_page(location)
  File "/usr/local/lib/python2.7/site-packages/pipenv/patched/notpip/index.py", line 719, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/usr/local/lib/python2.7/site-packages/pipenv/patched/notpip/index.py", line 831, in get_page
    resp.raise_for_status()
  File "/usr/local/lib/python2.7/site-packages/pipenv/../pipenv/../pip/_vendor/requests/models.py", line 862, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
pip._vendor.requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://testpypi.python.org/pypi/pipenv-test-public-package/

/usr/local/lib/python2.7/site-packages/pipenv/utils.py:1157: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/k_/yfvkl0_n2y9_t7xr9gd16r740000gn/T/pipenv-LtrMA1-requirements'>
  warnings.warn(warn_message, ResourceWarning)

Steps to reproduce

Using pipenv version 1.15.2, run pipenv install using the following Pipfile

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://testpypi.python.org/pypi"
verify_ssl = true
name = "testpypi"

[packages]
pipenv-test-private-package = "*"

---

For simplicity’s sake, I’ve hosted the contents of the private and public packages at the following gists:

• Private: https://gist.github.com/ryanwilsonperkin/585250b0558e7fbc380536c40d0b9005
• Public: https://gist.github.com/ryanwilsonperkin/d8d591e08e0996414eea27b25f4bb68c

My python -m pipenv.help is:

Pipenv version: '11.5.2'

Pipenv location: '/usr/local/lib/python2.7/site-packages/pipenv'

Python location: '/usr/local/opt/python/bin/python2.7'

Other Python installations in PATH:

• 2.6: /usr/bin/python2.6

• 2.6: /usr/bin/python2.6

• 2.7: /usr/local/bin/python2.7

• 2.7: /usr/local/bin/python2.7

• 2.7: /usr/bin/python2.7

• 3.6: /usr/local/bin/python3.6m

• 3.6: /usr/local/bin/python3.6

• 2.7.13: /usr/local/bin/python

• 2.7.10: /usr/bin/python

• 2.7.13: /usr/local/bin/python2

• 3.6.4: /usr/local/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '0',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '15.6.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 15.6.0: Mon Nov 13 21:58:35 PST 2017; root:xnu-3248.72.11~1/RELEASE_X86_64',
 'python_full_version': '2.7.13',
 'python_version': '2.7',
 'sys_platform': 'darwin'}

System environment variables:

• TERM_PROGRAM_VERSION
• LOGNAME
• USER
• NVM_DIR
• HOME
• PATH
• PYTHONUNBUFFERED
• DISPLAY
• TERM_PROGRAM
• LANG
• TERM
• Apple_PubSub_Socket_Render
• COLORFGBG
• SHLVL
• SECURITYSESSIONID
• XPC_FLAGS
• ITERM_SESSION_ID
• EDITOR
• TERM_SESSION_ID
• XPC_SERVICE_NAME
• SSH_AUTH_SOCK
• SHELL
• ITERM_PROFILE
• PIP_PYTHON_PATH
• TMPDIR
• __CF_USER_TEXT_ENCODING
• PWD
• COLORTERM

[techalchemy]

I think we understand the issue. Here is my proposed solution, if there are any concerns raise them here and we can discuss:

Our current implementation iterates over the indexes and attempts to install each top level package using each index. It used to fail silently, obviously that doesn’t work anymore. This is problematic for a couple of reasons

1. We have to manually iterate over each package and each index unless it’s explicitly stated
2. We don’t resolve dependencies across indexes (i.e. if they are not on the same index, or if a dependency is on a private index)

My proposal is this:

1. The first listed source will become the primary index
2. The subsequent sources will become arguments to EXTRA_INDEX_URL
3. Perhaps we can add an optional flag that indicates that the location should be treated as a FIND_LINKS location rather than an EXTRA_INDEX_URL which would enable local wheel repositories and the like

Thoughts?