pipenv not honoring setup_requires while locking
See original GitHub issueIssue description
For setup.py’s which have setup_requires
like cython
, pipenv does not seem to be honoring it while locking the Pipfile
Expected result
Should be able to lock with setup.py requirements
Actual result
alex@amohr-C02C14YVMD6M /tmp/foo $ cat Pipfile
[[source]]
name = "pypi"
url = "https://pypi.fbn.org/simple"
verify_ssl = true
[packages]
ddtrace = {extras = ["profiling"], version = "==0.36.1" }
[requires]
python_version = "3.8"
alex@amohr-C02C14YVMD6M /tmp/foo $ pipenv lock --verbose
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Building requirements...
Resolving dependencies...
⠙ Locking...
✘ Locking Failed!
ROUND 1
Current constraints:
ddtrace[profiling]==0.36.1 (from -r /var/folders/my/1cvd396x2m35py37s4lfhjv80000gp/T/pipenvsa2yqfuyrequirements/pipenv-qgladdka-constraints.txt (line 2))
Finding the best candidates:
found candidate ddtrace[profiling]==0.36.1 (constraint was ==0.36.1)
Finding secondary dependencies:
ddtrace[profiling]==0.36.1 not in cache, need to check index
ERROR:pip.subprocessor:Command errored out with exit status 1:
command: /Users/alex/.local/share/virtualenvs/foo-mzLZXOua/bin/python /usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/my/1cvd396x2m35py37s4lfhjv80000gp/T/tmp7_75fjjs
cwd: /private/var/folders/my/1cvd396x2m35py37s4lfhjv80000gp/T/pip-resolver-uciotq_i/ddtrace
Complete output (14 lines):
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 257, in <module>
main()
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 240, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 110, in prepare_metadata_for_build_wheel
return hook(metadata_directory, config_settings)
File "/Users/alex/.local/share/virtualenvs/foo-mzLZXOua/lib/python3.8/site-packages/setuptools/build_meta.py", line 158, in prepare_metadata_for_build_wheel
self.run_setup()
File "/Users/alex/.local/share/virtualenvs/foo-mzLZXOua/lib/python3.8/site-packages/setuptools/build_meta.py", line 143, in run_setup
exec(compile(code, __file__, 'exec'), locals())
File "setup.py", line 9, in <module>
from Cython.Build import cythonize # noqa: I100
ModuleNotFoundError: No module named 'Cython'
----------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 807, in <module>
main()
File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 802, in main
_main(parsed.pre, parsed.clear, parsed.verbose, parsed.system, parsed.write,
File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 785, in _main
resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages)
File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 746, in resolve_packages
results, resolver = resolve(
File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 728, in resolve
return resolve_deps(
File "/usr/local/lib/python3.8/site-packages/pipenv/utils.py", line 1371, in resolve_deps
results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
File "/usr/local/lib/python3.8/site-packages/pipenv/utils.py", line 1086, in actually_resolve_deps
resolver.resolve()
File "/usr/local/lib/python3.8/site-packages/pipenv/utils.py", line 807, in resolve
results = self.resolver.resolve(max_rounds=environments.PIPENV_MAX_ROUNDS)
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 180, in resolve
has_changed, best_matches = self._resolve_one_round()
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 268, in _resolve_one_round
their_constraints.extend(self._iter_dependencies(best_match))
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 383, in _iter_dependencies
dependencies = self.repository.get_dependencies(ireq)
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 226, in get_dependencies
legacy_results = self.get_legacy_dependencies(ireq)
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 347, in get_legacy_dependencies
results, ireq = self.resolve_reqs(
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 303, in resolve_reqs
results = resolver._resolve_one(reqset, ireq)
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/legacy_resolve.py", line 339, in _resolve_one
abstract_dist = self._get_abstract_dist_for(req_to_install)
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/legacy_resolve.py", line 287, in _get_abstract_dist_for
abstract_dist = self.preparer.prepare_linked_requirement(req)
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 508, in prepare_linked_requirement
abstract_dist = _get_prepared_distribution(
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 95, in _get_prepared_distribution
abstract_dist.prepare_distribution_metadata(finder, build_isolation)
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/distributions/sdist.py", line 40, in prepare_distribution_metadata
self.req.prepare_metadata()
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/req/req_install.py", line 564, in prepare_metadata
self.metadata_directory = self._generate_metadata()
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/req/req_install.py", line 549, in _generate_metadata
return generate_metadata(
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/operations/build/metadata.py", line 36, in generate_metadata
distinfo_dir = backend.prepare_metadata_for_build_wheel(
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/wrappers.py", line 167, in prepare_metadata_for_build_wheel
return self._call_hook('prepare_metadata_for_build_wheel', {
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/wrappers.py", line 245, in _call_hook
self._subprocess_runner(
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/utils/subprocess.py", line 271, in runner
call_subprocess(
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/utils/subprocess.py", line 242, in call_subprocess
raise InstallationError(exc_msg)
pipenv.patched.notpip._internal.exceptions.InstallationError: Command errored out with exit status 1: /Users/alex/.local/share/virtualenvs/foo-mzLZXOua/bin/python /usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/my/1cvd396x2m35py37s4lfhjv80000gp/T/tmp7_75fjjs Check the logs for full command output.
ROUND 1
Current constraints:
ddtrace[profiling]==0.36.1 (from -r /var/folders/my/1cvd396x2m35py37s4lfhjv80000gp/T/pipenvsa2yqfuyrequirements/pipenv-qgladdka-constraints.txt (line 2))
Finding the best candidates:
found candidate ddtrace[profiling]==0.36.1 (constraint was ==0.36.1)
Finding secondary dependencies:
ddtrace[profiling]==0.36.1 not in cache, need to check index
ERROR:pip.subprocessor:Command errored out with exit status 1:
command: /Users/alex/.local/share/virtualenvs/foo-mzLZXOua/bin/python /usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/my/1cvd396x2m35py37s4lfhjv80000gp/T/tmp7_75fjjs
cwd: /private/var/folders/my/1cvd396x2m35py37s4lfhjv80000gp/T/pip-resolver-uciotq_i/ddtrace
Complete output (14 lines):
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 257, in <module>
main()
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 240, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 110, in prepare_metadata_for_build_wheel
return hook(metadata_directory, config_settings)
File "/Users/alex/.local/share/virtualenvs/foo-mzLZXOua/lib/python3.8/site-packages/setuptools/build_meta.py", line 158, in prepare_metadata_for_build_wheel
self.run_setup()
File "/Users/alex/.local/share/virtualenvs/foo-mzLZXOua/lib/python3.8/site-packages/setuptools/build_meta.py", line 143, in run_setup
exec(compile(code, __file__, 'exec'), locals())
File "setup.py", line 9, in <module>
from Cython.Build import cythonize # noqa: I100
ModuleNotFoundError: No module named 'Cython'
----------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 807, in <module>
main()
File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 802, in main
_main(parsed.pre, parsed.clear, parsed.verbose, parsed.system, parsed.write,
File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 785, in _main
resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages)
File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 746, in resolve_packages
results, resolver = resolve(
File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 728, in resolve
return resolve_deps(
File "/usr/local/lib/python3.8/site-packages/pipenv/utils.py", line 1371, in resolve_deps
results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
File "/usr/local/lib/python3.8/site-packages/pipenv/utils.py", line 1086, in actually_resolve_deps
resolver.resolve()
File "/usr/local/lib/python3.8/site-packages/pipenv/utils.py", line 807, in resolve
results = self.resolver.resolve(max_rounds=environments.PIPENV_MAX_ROUNDS)
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 180, in resolve
has_changed, best_matches = self._resolve_one_round()
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 268, in _resolve_one_round
their_constraints.extend(self._iter_dependencies(best_match))
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 383, in _iter_dependencies
dependencies = self.repository.get_dependencies(ireq)
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 226, in get_dependencies
legacy_results = self.get_legacy_dependencies(ireq)
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 347, in get_legacy_dependencies
results, ireq = self.resolve_reqs(
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 303, in resolve_reqs
results = resolver._resolve_one(reqset, ireq)
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/legacy_resolve.py", line 339, in _resolve_one
abstract_dist = self._get_abstract_dist_for(req_to_install)
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/legacy_resolve.py", line 287, in _get_abstract_dist_for
abstract_dist = self.preparer.prepare_linked_requirement(req)
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 508, in prepare_linked_requirement
abstract_dist = _get_prepared_distribution(
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 95, in _get_prepared_distribution
abstract_dist.prepare_distribution_metadata(finder, build_isolation)
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/distributions/sdist.py", line 40, in prepare_distribution_metadata
self.req.prepare_metadata()
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/req/req_install.py", line 564, in prepare_metadata
self.metadata_directory = self._generate_metadata()
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/req/req_install.py", line 549, in _generate_metadata
return generate_metadata(
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/operations/build/metadata.py", line 36, in generate_metadata
distinfo_dir = backend.prepare_metadata_for_build_wheel(
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/wrappers.py", line 167, in prepare_metadata_for_build_wheel
return self._call_hook('prepare_metadata_for_build_wheel', {
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/wrappers.py", line 245, in _call_hook
self._subprocess_runner(
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/utils/subprocess.py", line 271, in runner
call_subprocess(
File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/utils/subprocess.py", line 242, in call_subprocess
raise InstallationError(exc_msg)
pipenv.patched.notpip._internal.exceptions.InstallationError: Command errored out with exit status 1: /Users/alex/.local/share/virtualenvs/foo-mzLZXOua/bin/python /usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/my/1cvd396x2m35py37s4lfhjv80000gp/T/tmp7_75fjjs Check the logs for full command output.
Steps to replicate
See Pipfile above and lock command run
$ pipenv --support
Pipenv version: '2020.4.1b1'
Pipenv location: '/usr/local/lib/python3.8/site-packages/pipenv'
Python location: '/usr/local/opt/python@3.8/bin/python3.8'
Python installations found:
3.8.2
:/usr/local/opt/python@3.8/bin/python3
3.8.2
:/usr/local/opt/python@3.8/bin/python3.8
3.8.0
:/usr/local/bin/python3.8
3.7.3
:/usr/bin/python3
3.7.0
:/usr/local/bin/python3
3.7.0
:/usr/local/bin/python3.7m
3.7.0
:/usr/local/bin/python3.7
3.6.7
:/usr/local/bin/python3.6
3.6.7
:/usr/local/bin/python3.6m
3.5.9
:/Users/alex/.pyenv/versions/3.5.9/bin/python3
2.7.16
:/usr/bin/python2
2.7.16
:/usr/bin/python2.7
2.7.15
:/usr/local/bin/python2
2.7.15
:/usr/local/bin/python2.7
PEP 508 Information:
{'implementation_name': 'cpython',
'implementation_version': '3.8.2',
'os_name': 'posix',
'platform_machine': 'x86_64',
'platform_python_implementation': 'CPython',
'platform_release': '19.4.0',
'platform_system': 'Darwin',
'platform_version': 'Darwin Kernel Version 19.4.0: Wed Mar 4 22:28:40 PST '
'2020; root:xnu-6153.101.6~15/RELEASE_X86_64',
'python_full_version': '3.8.2',
'python_version': '3.8',
'sys_platform': 'darwin'}
System environment variables:
SHELL
XPC_FLAGS
HISTCONTROL
TERM_PROGRAM_VERSION
PKG_CONFIG_PATH
HISTSIZE
HDF5_DIR
SSH_AUTH_SOCK
TERM_SESSION_ID
GPG_TTY
serverFlavor
PWD
LOGNAME
MANPATH
LaunchInstanceID
HOME
LANG
SECURITYSESSIONID
TMPDIR
CLICOLOR
FBNSECRETS_FORCE_USER
OPENSC_LIBS
NVM_DIR
TERM
USER
DISPLAY
SHLVL
XPC_SERVICE_NAME
HOMEBREW_GITHUB_API_TOKEN
PS1
PATH
OLDPWD
GOPATH
TERM_PROGRAM
_
__CF_USER_TEXT_ENCODING
PIP_DISABLE_PIP_VERSION_CHECK
PYTHONDONTWRITEBYTECODE
PIP_SHIMS_BASE_MODULE
PIP_PYTHON_PATH
PYTHONFINDER_IGNORE_UNSUPPORTED
Pipenv–specific environment variables:
Debug–specific environment variables:
PATH
:~/.local/bin:/usr/local/opt/python@3.8/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Library/Apple/usr/bin:/Users/alex/bin:/usr/local/opt/go/libexec/bin:/Users/alex/gowork/bin:/Applications/Postgres.app/Contents/Versions/9.6/bin:/usr/local/bin:/usr/local/sbin:/Users/alex/Downloads/google-cloud-sdk/bin:/Users/alex/dev/fbn.com/ops/docker/ecr/bin
SHELL
:/usr/local/bin/bash
LANG
:en_US.UTF-8
PWD
:/tmp/foo
Contents of Pipfile
(‘/private/tmp/foo/Pipfile’):
[[source]]
name = "pypi"
url = "https://pypi.fbn.org/simple"
verify_ssl = true
[packages]
ddtrace = {extras = ["profiling"], version = "==0.36.1" }
[requires]
python_version = "3.8"
Issue Analytics
- State:
- Created 3 years ago
- Reactions:1
- Comments:19 (11 by maintainers)
Top Results From Across the Web
Common Pipenv Errors and How to Solve Them: Why Won't it ...
Pipfile won't lock when it tries to install packages that don't exist, i.e., typos. ... Pipenv looks for the package you specify, whether...
Read more >How to fix locking failed in pipenv? - Stack Overflow
It seems like a library is requiring a version >= importlib-metadata 2.0 . When I pinned the following dependencies it didn't work at...
Read more >pipenv Documentation - Read the Docs
Generates and checks file hashes for locked dependencies when installing from ... Note: Pipenv is designed to be used by non-privileged OS users....
Read more >Advanced Usage of Pipenv — pipenv 7.7.9 documentation
Here is a simple example of a Pipfile and the resulting Pipfile.lock . ... have a requirements.txt file available when running pipenv install...
Read more >pipenv Changelog - pyup.io
and also fix regression where lock phase did not update the hash value. ... Use ``packages`` as contraints when locking ``dev-packages`` in Pipfile....
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Hey so I have a more thorough grasp of this issue now and there are actually two separate issues in play which led me down a few confusing paths before sorting this out.
Firstly, the issue with
lazy-object-proxy
– the package in question has asetup.py
which takes advantage ofsetuptools-scm
(other issues may come up with regard to this because it probably doesn’t work properly when resolving sdists either), as well as having apyproject.toml
withbuild-system.requires
defined, but nobuild-system.build-backend
.In our resolution step,
pip
was attempting to set thebuild-system.build-backend
tosetuptools.build_meta:__legacy__
which, if you’ve had your virtualenv around and never updated setuptools, might not exist. This normally doesn’t matter, sincepip
uses its own internalpep517
build backend and builds in an isolate build environment.In the case of
ddtrace
, there is also apyproject.toml
file, which, in this case, includes a sufficiently unconstrainedbuild-system.requires
and explicitly setsbuild-system.build-backend
tosetuptools.build_meta
. Insetup.py
, however, it importsCython
, one of itsbuild-system.requires
items. In another issue I speculated that we might fix our dependency resolution challenges if we enabled build isolation during dependency resolution.Following up on that, I discovered that when we are determining whether to build in isolation or use
pep517
, we are doing this a bit…wrong.In this first bit of resolver code, you can see that we were relying on truthy values of
use_pep517
andbuild_isolation
rather than the more strongis False
comparison – i.e., ifis_pep517 is None
, we would still decide to pass--no-use-pep517
to pip.And here where we decide what actually gets passed in, we were doing some overly complicated logical expressions which wind up evaluating in an undesirable way. To break the issue down:
use_pep517 = False if (condition1) else (True if condition2 else None)
. So, even if this accurately evaluated toNone
, we still read that as--no-<x>
.As a result I’ve provided a more general function for reading these types of values from the environment in the linked PR and a way to build in isolation by default. I’m a little bit unsure about whether this might break certain workflows so I want to consider whether we need any additional fallbacks for those cases.
That said, thanks for providing the detail required to troubleshoot this one, it was definitely tricky!
As my final note for the evening I am able to reproduce this by creating a virtualenv with an old version of
setuptools
(I initially tried<41.0
but that seemed to work ok, while the implicated version40.2.0
seems to be completely broken for these packages). I have just wrapped up some changes that will implement the functionality I described above, i.e. upgrading setuptools if pipenv bumps into this bug and re-resolving.After looking over my own libraries I’ve realized that where I handle resolution of packages with no backend specified and no build requirements, I begin by forcing
setuptools>=40.8
so I assume this is the version where the__legacy__
backend was added.Tomorrow morning I will throw together a fake package with a
setuptools.build_meta:__legacy__
backend and a quick test. If anyone foresees any issues with this approach, please raise them