0.7.1 cannot be installed on darwin/arm64 with Python 3.9

What happened: pip install “fastparquet==0.7.1”

Failed, because it tried to install numpy 1.20.0 (even if a newer version [1.21.2] was already present in the venv) which failed to build from sources. This is because:

• there is no prebuilt package for fastparquet 0.7.1 at PyPI for darwin/arm64/cp39
• and setup.py uses oldest-supported-numpy as a build dependency
• and oldest-supported-numpy is currently broken for darwin/arm64/cp39 because it points to a version (1.20.0) for which there is no prebuilt package at PyPI for the platform/python version combo
• and building numpy from sources on darwin/arm64 is notoriously fragile (and obviously broken for the numpy/OS/python version combo)

**Suggestion: Do not use the oldest-supported-numpy meta-package. It is a well meant, but unfortunately unsustainable approach. It results in requiring 1 specific version for an platform/python version combo which is likely to break frequently. Just requiring a minimum version of numpy for fastparquet has IMHO higher probability of success. In this specific case, it would’ve actually worked 😃.

What you expected to happen:

fasparquet 0.7.1. to be installed (without installing an older version of numpy).

Minimal Complete Verifiable Example: See above.

Environment:

• Dask version: N/A (I just use fastparquet with pandas)
• Python version: 3.9.5 (default, May 3 2021, 19:12:05) [Clang 12.0.5 (clang-1205.0.22.9)]
• Operating System:

ProductName:	macOS
ProductVersion:	11.4
BuildVersion:	20F71

• Install method (conda, pip, source): pip (which tries to build wheel package)

[cpaton8]

Hi all, I am also trying to install fastparquet using poetry via poetry add fastparquet which sources from PyPI and just wanted to confirm that I am also receiving an error regarding the numpy version (see below). numpy 1.21.2 is already installed when trying to add fastparquet.

error message details

EnvCommandError

Command [‘/Users/christinapaton/opt/miniconda3/envs/survey2/bin/pip’, ‘install’, ‘–no-deps’, ‘file:///Users/christinapaton/Library/Caches/pypoetry/artifacts/d7/2b/8e/09320fd2f81069fcc2682cccc7e084152fdedd4e4e9e09fd9c2c7a2910/fastparquet-0.7.1.tar.gz’] errored with the following return code 1, and output: Processing /Users/christinapaton/Library/Caches/pypoetry/artifacts/d7/2b/8e/09320fd2f81069fcc2682cccc7e084152fdedd4e4e9e09fd9c2c7a2910/fastparquet-0.7.1.tar.gz Building wheels for collected packages: fastparquet Building wheel for fastparquet (setup.py): started Building wheel for fastparquet (setup.py): finished with status ‘error’ ERROR: Command errored out with exit status 1: command: /Users/christinapaton/opt/miniconda3/envs/survey2/bin/python3.9 -u -c ‘import io, os, sys, setuptools, tokenize; sys.argv[0] = ‘"’"’/private/var/folders/_l/23fh_hz93hnb8jphrxrrl7dw0000gn/T/pip-req-build-9ud3vg6w/setup.py’“'”‘; file=’“'”‘/private/var/folders/_l/23fh_hz93hnb8jphrxrrl7dw0000gn/T/pip-req-build-9ud3vg6w/setup.py’“'”‘;f = getattr(tokenize, ‘"’“‘open’”’“‘, open)(file) if os.path.exists(file) else io.StringIO(’”‘“‘from setuptools import setup; setup()’”’“‘);code = f.read().replace(’”‘"’\r\n’“'”‘, ‘"’"’\n’“'”‘);f.close();exec(compile(code, file, ‘"’“‘exec’”’“‘))’ bdist_wheel -d /private/var/folders/_l/23fh_hz93hnb8jphrxrrl7dw0000gn/T/pip-wheel-rv0mi906 cwd: /private/var/folders/_l/23fh_hz93hnb8jphrxrrl7dw0000gn/T/pip-req-build-9ud3vg6w/ Complete output (27 lines): Traceback (most recent call last): File “<string>”, line 1, in <module> File “/private/var/folders/_l/23fh_hz93hnb8jphrxrrl7dw0000gn/T/pip-req-build-9ud3vg6w/setup.py”, line 47, in <module> setup( File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/setuptools/init.py”, line 152, in setup _install_setup_requires(attrs) File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/setuptools/init.py”, line 147, in _install_setup_requires dist.fetch_build_eggs(dist.setup_requires) File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/setuptools/dist.py”, line 806, in fetch_build_eggs resolved_dists = pkg_resources.working_set.resolve( File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/pkg_resources/init.py”, line 766, in resolve dist = best[req.key] = env.best_match( File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/pkg_resources/init.py”, line 1051, in best_match return self.obtain(req, installer) File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/pkg_resources/init.py”, line 1063, in obtain return installer(requirement) File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/setuptools/dist.py”, line 877, in fetch_build_egg return fetch_build_egg(self, req) File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/setuptools/installer.py”, line 80, in fetch_build_egg wheel.install_as_egg(dist_location) File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/setuptools/wheel.py”, line 95, in install_as_egg self._install_as_egg(destination_eggdir, zf) File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/setuptools/wheel.py”, line 103, in _install_as_egg self._convert_metadata(zf, destination_eggdir, dist_info, egg_info) File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/setuptools/wheel.py”, line 124, in _convert_metadata os.mkdir(destination_eggdir) FileExistsError: [Errno 17] File exists: ‘/private/var/folders/_l/23fh_hz93hnb8jphrxrrl7dw0000gn/T/pip-req-build-9ud3vg6w/.eggs/numpy-1.19.3-py3.9-macosx-10.9-x86_64.egg’ ---------------------------------------- ERROR: Failed building wheel for fastparquet Running setup.py clean for fastparquet ERROR: Command errored out with exit status 1: command: /Users/christinapaton/opt/miniconda3/envs/survey2/bin/python3.9 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '”‘"’/private/var/folders/_l/23fh_hz93hnb8jphrxrrl7dw0000gn/T/pip-req-build-9ud3vg6w/setup.py’“'”‘; file=’“'”‘/private/var/folders/_l/23fh_hz93hnb8jphrxrrl7dw0000gn/T/pip-req-build-9ud3vg6w/setup.py’“'”‘;f = getattr(tokenize, ‘"’“‘open’”’“‘, open)(file) if os.path.exists(file) else io.StringIO(’”‘“‘from setuptools import setup; setup()’”’“‘);code = f.read().replace(’”‘"’\r\n’“'”‘, ‘"’"’\n’“'”‘);f.close();exec(compile(code, file, ‘"’“‘exec’”’“‘))’ clean --all cwd: /private/var/folders/_l/23fh_hz93hnb8jphrxrrl7dw0000gn/T/pip-req-build-9ud3vg6w Complete output (27 lines): Traceback (most recent call last): File “<string>”, line 1, in <module> File “/private/var/folders/_l/23fh_hz93hnb8jphrxrrl7dw0000gn/T/pip-req-build-9ud3vg6w/setup.py”, line 47, in <module> setup( File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/setuptools/init.py”, line 152, in setup _install_setup_requires(attrs) File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/setuptools/init.py”, line 147, in _install_setup_requires dist.fetch_build_eggs(dist.setup_requires) File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/setuptools/dist.py”, line 806, in fetch_build_eggs resolved_dists = pkg_resources.working_set.resolve( File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/pkg_resources/init.py”, line 766, in resolve dist = best[req.key] = env.best_match( File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/pkg_resources/init.py”, line 1051, in best_match return self.obtain(req, installer) File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/pkg_resources/init.py”, line 1063, in obtain return installer(requirement) File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/setuptools/dist.py”, line 877, in fetch_build_egg return fetch_build_egg(self, req) File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/setuptools/installer.py”, line 80, in fetch_build_egg wheel.install_as_egg(dist_location) File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/setuptools/wheel.py”, line 95, in install_as_egg self._install_as_egg(destination_eggdir, zf) File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/setuptools/wheel.py”, line 103, in _install_as_egg self._convert_metadata(zf, destination_eggdir, dist_info, egg_info) File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/setuptools/wheel.py”, line 124, in _convert_metadata os.mkdir(destination_eggdir) FileExistsError: [Errno 17] File exists: ‘/private/var/folders/_l/23fh_hz93hnb8jphrxrrl7dw0000gn/T/pip-req-build-9ud3vg6w/.eggs/numpy-1.19.3-py3.9-macosx-10.9-x86_64.egg’ ---------------------------------------- ERROR: Failed cleaning build dir for fastparquet Failed to build fastparquet Installing collected packages: fastparquet Running setup.py install for fastparquet: started Running setup.py install for fastparquet: finished with status ‘error’ ERROR: Command errored out with exit status 1: command: /Users/christinapaton/opt/miniconda3/envs/survey2/bin/python3.9 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '”‘"’/private/var/folders/_l/23fh_hz93hnb8jphrxrrl7dw0000gn/T/pip-req-build-9ud3vg6w/setup.py’“'”‘; file=’“'”‘/private/var/folders/_l/23fh_hz93hnb8jphrxrrl7dw0000gn/T/pip-req-build-9ud3vg6w/setup.py’“'”‘;f = getattr(tokenize, ‘"’“‘open’”’“‘, open)(file) if os.path.exists(file) else io.StringIO(’”‘“‘from setuptools import setup; setup()’”’“‘);code = f.read().replace(’”‘"’\r\n’“'”‘, ‘"’"’\n’“'”‘);f.close();exec(compile(code, file, ‘"’“‘exec’”’“‘))’ install --record /private/var/folders/_l/23fh_hz93hnb8jphrxrrl7dw0000gn/T/pip-record-bcplr9pw/install-record.txt --single-version-externally-managed --compile --install-headers /Users/christinapaton/opt/miniconda3/envs/survey2/include/python3.9/fastparquet cwd: /private/var/folders/_l/23fh_hz93hnb8jphrxrrl7dw0000gn/T/pip-req-build-9ud3vg6w/ Complete output (27 lines): Traceback (most recent call last): File “<string>”, line 1, in <module> File “/private/var/folders/_l/23fh_hz93hnb8jphrxrrl7dw0000gn/T/pip-req-build-9ud3vg6w/setup.py”, line 47, in <module> setup( File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/setuptools/init.py”, line 152, in setup _install_setup_requires(attrs) File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/setuptools/init.py”, line 147, in _install_setup_requires dist.fetch_build_eggs(dist.setup_requires) File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/setuptools/dist.py”, line 806, in fetch_build_eggs resolved_dists = pkg_resources.working_set.resolve( File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/pkg_resources/init.py”, line 766, in resolve dist = best[req.key] = env.best_match( File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/pkg_resources/init.py”, line 1051, in best_match return self.obtain(req, installer) File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/pkg_resources/init.py”, line 1063, in obtain return installer(requirement) File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/setuptools/dist.py”, line 877, in fetch_build_egg return fetch_build_egg(self, req) File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/setuptools/installer.py”, line 80, in fetch_build_egg wheel.install_as_egg(dist_location) File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/setuptools/wheel.py”, line 95, in install_as_egg self._install_as_egg(destination_eggdir, zf) File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/setuptools/wheel.py”, line 103, in _install_as_egg self._convert_metadata(zf, destination_eggdir, dist_info, egg_info) File “/Users/christinapaton/opt/miniconda3/envs/survey2/lib/python3.9/site-packages/setuptools/wheel.py”, line 124, in _convert_metadata os.mkdir(destination_eggdir) FileExistsError: [Errno 17] File exists: ‘/private/var/folders/_l/23fh_hz93hnb8jphrxrrl7dw0000gn/T/pip-req-build-9ud3vg6w/.eggs/numpy-1.19.3-py3.9-macosx-10.9-x86_64.egg’ ---------------------------------------- ERROR: Command errored out with exit status 1: /Users/christinapaton/opt/miniconda3/envs/survey2/bin/python3.9 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '”‘"’/private/var/folders/_l/23fh_hz93hnb8jphrxrrl7dw0000gn/T/pip-req-build-9ud3vg6w/setup.py’“'”‘; file=’“'”‘/private/var/folders/_l/23fh_hz93hnb8jphrxrrl7dw0000gn/T/pip-req-build-9ud3vg6w/setup.py’“'”‘;f = getattr(tokenize, ‘"’“‘open’”’“‘, open)(file) if os.path.exists(file) else io.StringIO(’”‘“‘from setuptools import setup; setup()’”’“‘);code = f.read().replace(’”‘"’\r\n’“'”‘, ‘"’"’\n’“'”‘);f.close();exec(compile(code, file, ‘"’“‘exec’”’"‘))’ install --record /private/var/folders/_l/23fh_hz93hnb8jphrxrrl7dw0000gn/T/pip-record-bcplr9pw/install-record.txt --single-version-externally-managed --compile --install-headers /Users/christinapaton/opt/miniconda3/envs/survey2/include/python3.9/fastparquet Check the logs for full command output.

at ~/.poetry/lib/poetry/utils/env.py:1180 in run 1176│ output = subprocess.check_output( 1177│ cmd, stderr=subprocess.STDOUT, **kwargs 1178│ ) 1179│ except CalledProcessError as e: → 1180│ raise EnvCommandError(e, input=input) 1181│ 1182│ return decode(output) 1183│ 1184│ def execute(self, bin, *args, **kwargs):

[martindurant]

Let me summarise the problem for myself, to see if I understand the situation clearly.

• there are no wheels for fastparquet on your system, because at the last tag, cibuildtool was not configured or could not yet make them
• when asking pip to build from source, it picks a numpy version using oldest-supported-numpy, in order to build a binary that works on all current numpy versions
• for the specific version of python, oldest-supported-numpy pointed to a version which hasn’t been made a wheel on your OS either (and pip fails to build it from source)
• oldest-supported-numpy should have, in the meantime, been updated to something that does work.

Do I have it right?

I wonder, is conda a possible avenue for you, if only to create the wheel you need?