Pipenv>=2020 can't install local packages depending between them

Issue description

If a local package A has a dependency on another local package version B==X that is not present in the public pypi repository, the installation will fail.

The whole 2020.* series are affected. The 2022.1.8 version is also affected.

2018.11.26 was unaffected

Observations

• The fact that A is installed locally might not be relevant but it is hard to find a package that depends on a second package in a version that is not found in pypi.
• If the B package does not exist at all, the error message is about “No matching distribution found for B” instead of “There are incompatible versions in the resolved dependencies”

Expected result

The local folder containing the B==X package satisifes the A package dependency on it.

Actual result

(Edited)

Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Building requirements...
Resolving dependencies...
✘ Locking Failed! 
[ResolutionFailure]:   File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 785, in _main
[ResolutionFailure]:       resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages)
[ResolutionFailure]:   File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 746, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:   File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 728, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "/usr/local/lib/python3.8/site-packages/pipenv/utils.py", line 1378, in resolve_deps
[ResolutionFailure]:       results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]:   File "/usr/local/lib/python3.8/site-packages/pipenv/utils.py", line 1093, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/usr/local/lib/python3.8/site-packages/pipenv/utils.py", line 818, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
 Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: Could not find a version that matches B==X (from -r /tmp/pipenv4lmztgo3requirements/pipenv-7iw66_7i-constraints.txt (line 8))
Tried: [....]
There are incompatible versions in the resolved dependencies:

Steps to replicate

A minirepo is provided to replicate the issue

https://github.com/n1ngu/pipenv-issue-4553

Clone it and try to lock the dependencies:

pipenv lock --verbose

Output:

root@2ed6ff938352:/mnt# pipenv lock --verbose
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py:218: PipDeprecationWarning: DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.
 pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.
  deprecated(
/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py:218: PipDeprecationWarning: DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.
 pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.
  deprecated(
Resolving dependencies...
Reporter.starting()
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.starting()
Reporter.adding_requirement(SpecifierRequirement('nonexistent'), None)
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('nonexistent'), None)
CRITICAL:pipenv.patched.notpip._internal.resolution.resolvelib.factory:Could not find a version that satisfies the requirement nonexistent (from versions: none)
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/resolvelib/resolvers.py", line 341, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
    raise RequirementsConflicted(criterion)
pipenv.patched.notpip._vendor.resolvelib.resolvers.RequirementsConflicted: Requirements conflict: SpecifierRequirement('nonexistent')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/resolution/resolvelib/resolver.py", line 94, in resolve
    result = self._result = resolver.resolve(
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/resolvelib/resolvers.py", line 472, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/resolvelib/resolvers.py", line 343, in resolve
    raise ResolutionImpossible(e.criterion.information)
pipenv.patched.notpip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('nonexistent'), parent=None)]
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pipenv/utils.py", line 882, in resolve
    results = resolver.resolve(self.constraints, check_supported_wheels=False)
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/resolution/resolvelib/resolver.py", line 103, in resolve
    raise error from e
pipenv.patched.notpip._internal.exceptions.DistributionNotFound: No matching distribution found for nonexistent
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pipenv/resolver.py", line 766, in <module>
    main()
  File "/usr/local/lib/python3.9/site-packages/pipenv/resolver.py", line 760, in main
    _main(parsed.pre, parsed.clear, parsed.verbose, parsed.system, parsed.write,
  File "/usr/local/lib/python3.9/site-packages/pipenv/resolver.py", line 743, in _main
    resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
  File "/usr/local/lib/python3.9/site-packages/pipenv/resolver.py", line 704, in resolve_packages
    results, resolver = resolve(
  File "/usr/local/lib/python3.9/site-packages/pipenv/resolver.py", line 685, in resolve
    return resolve_deps(
  File "/usr/local/lib/python3.9/site-packages/pipenv/utils.py", line 1377, in resolve_deps
    results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
  File "/usr/local/lib/python3.9/site-packages/pipenv/utils.py", line 1106, in actually_resolve_deps
    resolver.resolve()
  File "/usr/local/lib/python3.9/site-packages/pipenv/utils.py", line 884, in resolve
    raise ResolutionFailure(message=str(e))
pipenv.exceptions.ResolutionFailure: ERROR: No matching distribution found for nonexistent
✘ Locking Failed! 

---

$ pipenv --support

Pipenv version: '2022.1.8'

Pipenv location: '/usr/local/lib/python3.9/site-packages/pipenv'

Python location: '/usr/local/bin/python'

Python installations found:

• 3.9.10: /usr/local/bin/python
• 3.9.10: /usr/local/bin/python3.9
• 3.9.10: /usr/local/bin/python3
• 3.9.2: /usr/bin/python3.9
• 3.9.2: /usr/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.9.10',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.9.16-1-MANJARO',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP PREEMPT Mon Dec 21 22:00:46 UTC 2020',
 'python_full_version': '3.9.10',
 'python_version': '3.9',
 'sys_platform': 'linux'}

System environment variables:

• HOSTNAME
• PYTHON_VERSION
• PWD
• PYTHON_SETUPTOOLS_VERSION
• HOME
• LANG
• GPG_KEY
• TERM
• SHLVL
• PYTHON_PIP_VERSION
• PYTHON_GET_PIP_SHA256
• PS1
• PYTHON_GET_PIP_URL
• PATH
• _
• PIP_SHIMS_BASE_MODULE
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONDONTWRITEBYTECODE
• PIP_PYTHON_PATH
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
• LANG: C.UTF-8
• PWD: /mnt

---

Contents of Pipfile (‘/mnt/Pipfile’):

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
nonexistent = {path = "./nonexistent"}
foo = {path = "./foo"}

[requires]
python_version = "3.9"

[matteius]

@n1ngu It works in 2022.8.19 if you mark the packages as editable in your Pipfile:

matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-issue-4553$ git diff
diff --git a/Pipfile b/Pipfile
index 1f91e94..12867d4 100644
--- a/Pipfile
+++ b/Pipfile
@@ -6,8 +6,8 @@ verify_ssl = true
 [dev-packages]
 
 [packages]
-nonexistent = {path = "./nonexistent"}
-foo = {path = "./foo"}
+nonexistent = {editable = true, path = "./nonexistent"}
+foo = {editable = true, path = "./foo"}
 
 [requires]
 python_version = "3.9"

[matteius]

@n1ngu So at first look, and some background is that the locking/resolution/installation phase has all ben converted over to use pip internals. So with pip just trying to install package foo in your example is an issue because it cannot resolve the nonexistent package from the setup.cfg. It works in pip if I manually install pip install -e nonexistent/ first and then pip install -e foo/ … I think what is happening in newer versions of pipenv is that all requirements get passed to the resolver in one go and it is just building the local dependencies in isolation so when it gets to foo, even if nonexistent was already installed and in the lockfile, it search pypi and errors out.