Issue with NumPy, macOS 11 Big Sur, Python 3.9.1 Does pipenv not use the latest pip?
See original GitHub issuePeople are probably already aware of this issue, but adding my $0.02 as to what I think may be going on.
In a nutshell: NumPy fails to build on OS 11 and python 3.9.1. The error message is attached below, but NumPy is installed and works. (you can import numpy as np
from a python interpreter) However, if you try to import numpy in a notebook, you get an error message saying that numpy is using a buggy “accelerate” backend.
The error about the backend occurs when numpy is being built from source incorrectly instead of using the provided wheel. That known issue was supposedly fixed in a pip upgrade (20.3). However, I get the same error below, even when I have updated pip to 20.3.
However, when i create a new pipenv environment, it defaults to pip 20.2. So my guess is that even if I subsequently update pip to 20.3, that pipenv install numpy
is using the old version of pip.
Can anybody confirm and/or offer a workaround?
(blog) shane@Olympus ~/github/blog dev ● pipenv install numpy
Installing numpy...
Adding numpy to Pipfile's [packages]...
✔ Installation Succeeded
Pipfile.lock (16c839) out of date, updating to (49bf85)...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed!
ERROR:pip.subprocessor:Command errored out with exit status 1:
command: /Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/bin/python /usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/tmpk03mt0_f
cwd: /private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy
Complete output (209 lines):
Running from numpy source directory.
setup.py:480: UserWarning: Unrecognized setuptools command, proceeding with generating Cython sources and expanding templates
run_build = parse_setuppy_commands()
Cythonizing sources
Processing numpy/random/_bounded_integers.pxd.in
Processing numpy/random/_philox.pyx
Processing numpy/random/_bounded_integers.pyx.in
Processing numpy/random/_sfc64.pyx
Processing numpy/random/_mt19937.pyx
Processing numpy/random/bit_generator.pyx
Processing numpy/random/mtrand.pyx
Processing numpy/random/_generator.pyx
Processing numpy/random/_pcg64.pyx
Processing numpy/random/_common.pyx
blas_opt_info:
blas_mkl_info:
customize UnixCCompiler
libraries mkl_rt not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
NOT AVAILABLE
blis_info:
libraries blis not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
NOT AVAILABLE
openblas_info:
libraries openblas not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
NOT AVAILABLE
atlas_3_10_blas_threads_info:
Setting PTATLAS=ATLAS
libraries tatlas not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
NOT AVAILABLE
atlas_3_10_blas_info:
libraries satlas not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
NOT AVAILABLE
atlas_blas_threads_info:
Setting PTATLAS=ATLAS
libraries ptf77blas,ptcblas,atlas not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
NOT AVAILABLE
atlas_blas_info:
libraries f77blas,cblas,atlas not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
NOT AVAILABLE
accelerate_info:
libraries accelerate not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
Library accelerate was not found. Ignoring
libraries veclib not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
Library veclib was not found. Ignoring
FOUND:
extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]
FOUND:
extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]
non-existing path in 'numpy/distutils': 'site.cfg'
lapack_opt_info:
lapack_mkl_info:
libraries mkl_rt not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
NOT AVAILABLE
openblas_lapack_info:
libraries openblas not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
NOT AVAILABLE
openblas_clapack_info:
libraries openblas,lapack not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
NOT AVAILABLE
flame_info:
libraries flame not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
NOT AVAILABLE
atlas_3_10_threads_info:
Setting PTATLAS=ATLAS
libraries lapack_atlas not found in /Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib
libraries tatlas,tatlas not found in /Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib
libraries lapack_atlas not found in /usr/local/lib
libraries tatlas,tatlas not found in /usr/local/lib
libraries lapack_atlas not found in /usr/lib
libraries tatlas,tatlas not found in /usr/lib
<class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
NOT AVAILABLE
atlas_3_10_info:
libraries lapack_atlas not found in /Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib
libraries satlas,satlas not found in /Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib
libraries lapack_atlas not found in /usr/local/lib
libraries satlas,satlas not found in /usr/local/lib
libraries lapack_atlas not found in /usr/lib
libraries satlas,satlas not found in /usr/lib
<class 'numpy.distutils.system_info.atlas_3_10_info'>
NOT AVAILABLE
atlas_threads_info:
Setting PTATLAS=ATLAS
libraries lapack_atlas not found in /Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib
libraries ptf77blas,ptcblas,atlas not found in /Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib
libraries lapack_atlas not found in /usr/local/lib
libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
libraries lapack_atlas not found in /usr/lib
libraries ptf77blas,ptcblas,atlas not found in /usr/lib
<class 'numpy.distutils.system_info.atlas_threads_info'>
NOT AVAILABLE
atlas_info:
libraries lapack_atlas not found in /Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib
libraries f77blas,cblas,atlas not found in /Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib
libraries lapack_atlas not found in /usr/local/lib
libraries f77blas,cblas,atlas not found in /usr/local/lib
libraries lapack_atlas not found in /usr/lib
libraries f77blas,cblas,atlas not found in /usr/lib
<class 'numpy.distutils.system_info.atlas_info'>
NOT AVAILABLE
FOUND:
extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]
/usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py:274: UserWarning: Unknown distribution option: 'define_macros'
warnings.warn(msg)
running dist_info
running build_src
build_src
building py_modules sources
creating build
creating build/src.macosx-11-x86_64-3.9
creating build/src.macosx-11-x86_64-3.9/numpy
creating build/src.macosx-11-x86_64-3.9/numpy/distutils
building library "npymath" sources
Traceback (most recent call last):
File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 257, in <module>
main()
File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 240, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 110, in prepare_metadata_for_build_wheel
return hook(metadata_directory, config_settings)
File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-build-env-5hhkc_mx/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 157, in prepare_metadata_for_build_wheel
self.run_setup()
File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-build-env-5hhkc_mx/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 248, in run_setup
super(_BuildMetaLegacyBackend,
File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-build-env-5hhkc_mx/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 142, in run_setup
exec(compile(code, __file__, 'exec'), locals())
File "setup.py", line 508, in <module>
setup_package()
File "setup.py", line 500, in setup_package
setup(**metadata)
File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/core.py", line 169, in setup
return old_setup(**new_attr)
File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-build-env-5hhkc_mx/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 165, in setup
return distutils.core.setup(**attrs)
File "/usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 966, in run_commands
self.run_command(cmd)
File "/usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-build-env-5hhkc_mx/overlay/lib/python3.9/site-packages/setuptools/command/dist_info.py", line 31, in run
egg_info.run()
File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/command/egg_info.py", line 24, in run
self.run_command("build_src")
File "/usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/command/build_src.py", line 144, in run
self.build_sources()
File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/command/build_src.py", line 155, in build_sources
self.build_library_sources(*libname_info)
File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/command/build_src.py", line 288, in build_library_sources
sources = self.generate_sources(sources, (lib_name, build_info))
File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/command/build_src.py", line 378, in generate_sources
source = func(extension, build_dir)
File "numpy/core/setup.py", line 658, in get_mathlib_info
st = config_cmd.try_link('int main(void) { return 0;}')
File "/usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/config.py", line 241, in try_link
self._check_compiler()
File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/command/config.py", line 80, in _check_compiler
self.fcompiler = new_fcompiler(compiler=self.fcompiler,
File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/fcompiler/__init__.py", line 880, in new_fcompiler
compiler = get_default_fcompiler(plat, requiref90=requiref90,
File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/fcompiler/__init__.py", line 851, in get_default_fcompiler
compiler_type = _find_existing_fcompiler(matching_compiler_types,
File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/fcompiler/__init__.py", line 802, in _find_existing_fcompiler
c.customize(dist)
File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/fcompiler/__init__.py", line 526, in customize
linker_so_flags = self.flag_vars.linker_so
File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/fcompiler/environment.py", line 37, in __getattr__
return self._get_var(name, conf_desc)
File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/fcompiler/environment.py", line 53, in _get_var
var = self._hook_handler(name, hook)
File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/fcompiler/__init__.py", line 705, in _environment_hook
return hook()
File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/fcompiler/gnu.py", line 346, in get_flags_linker_so
flags = GnuFCompiler.get_flags_linker_so(self)
File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/fcompiler/gnu.py", line 136, in get_flags_linker_so
os.environ['MACOSX_DEPLOYMENT_TARGET'] = target
File "/usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/os.py", line 684, in __setitem__
value = self.encodevalue(value)
File "/usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/os.py", line 756, in encode
raise TypeError("str expected, not %s" % type(value).__name__)
TypeError: str expected, not int
----------------------------------------
Traceback (most recent call last):
File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/resolver.py", line 764, in <module>
main()
File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/resolver.py", line 758, in main
_main(parsed.pre, parsed.clear, parsed.verbose, parsed.system, parsed.write,
File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/resolver.py", line 741, in _main
resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/resolver.py", line 702, in resolve_packages
results, resolver = resolve(
File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/resolver.py", line 684, in resolve
return resolve_deps(
File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/utils.py", line 1395, in resolve_deps
results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/utils.py", line 1108, in actually_resolve_deps
resolver.resolve()
File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/utils.py", line 823, in resolve
results = self.resolver.resolve(max_rounds=environments.PIPENV_MAX_ROUNDS)
File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/piptools/resolver.py", line 180, in resolve
has_changed, best_matches = self._resolve_one_round()
File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/piptools/resolver.py", line 268, in _resolve_one_round
their_constraints.extend(self._iter_dependencies(best_match))
File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/piptools/resolver.py", line 383, in _iter_dependencies
dependencies = self.repository.get_dependencies(ireq)
File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 229, in get_dependencies
legacy_results = self.get_legacy_dependencies(ireq)
File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 350, in get_legacy_dependencies
results, ireq = self.resolve_reqs(
File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 306, in resolve_reqs
results = resolver._resolve_one(reqset, ireq)
File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/legacy_resolve.py", line 339, in _resolve_one
abstract_dist = self._get_abstract_dist_for(req_to_install)
File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/legacy_resolve.py", line 287, in _get_abstract_dist_for
abstract_dist = self.preparer.prepare_linked_requirement(req)
File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 508, in prepare_linked_requirement
abstract_dist = _get_prepared_distribution(
File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 95, in _get_prepared_distribution
abstract_dist.prepare_distribution_metadata(finder, build_isolation)
File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/distributions/sdist.py", line 40, in prepare_distribution_metadata
self.req.prepare_metadata()
File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/req/req_install.py", line 564, in prepare_metadata
self.metadata_directory = self._generate_metadata()
File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/req/req_install.py", line 549, in _generate_metadata
return generate_metadata(
File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/operations/build/metadata.py", line 36, in generate_metadata
distinfo_dir = backend.prepare_metadata_for_build_wheel(
File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/pep517/wrappers.py", line 167, in prepare_metadata_for_build_wheel
return self._call_hook('prepare_metadata_for_build_wheel', {
File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/pep517/wrappers.py", line 245, in _call_hook
self._subprocess_runner(
File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/utils/subprocess.py", line 271, in runner
call_subprocess(
File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/utils/subprocess.py", line 242, in call_subprocess
raise InstallationError(exc_msg)
pipenv.patched.notpip._internal.exceptions.InstallationError: Command errored out with exit status 1: /Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/bin/python /usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/tmpk03mt0_f Check the logs for full command output.
Issue Analytics
- State:
- Created 3 years ago
- Reactions:15
- Comments:29 (8 by maintainers)
Top GitHub Comments
Hi, this helped me:
export SYSTEM_VERSION_COMPAT=1
Source: https://stackoverflow.com/questions/63972113/big-sur-clang-invalid-version-error-due-to-macosx-deployment-target
A lot of work has gone into making macOS Big Sur work, but it requires the latest dependencies. Pipenv was released without the “bump all dependencies” step described in its docs, so sadly, it still doesn’t understand that “11.0” is a valid macOS version. If you set
export SYSTEM_VERSION_COMPAT=1
, then macOS will report itself as “10.16” instead, which temporarily fixes the problem (only on Intel machines, I believe).