Cython wheel not used but numpy wheel is

• Operating System: Alpine Linux 3.13
• Python version: 3.8.8
• Where Python was acquired: Alpine Package Registry
• h5py version: 3.2.1

I am building an h5py wheel in an Alpine build image. I have the wheels for numpy and Cython copied into the /wheels directory. During the pip wheel of h5py it correctly finds the numpy wheel in the wheels directory but does not seem to care about the Cython wheel.

Any thoughts on this? It doesn’t look like the build requirements list Cython as a requirement but I may be missing something.

/ # ls -lA /wheels
total 6428
-rw-r--r--    1 root     root        980535 Apr 11 01:13 Cython-0.29.22-py2.py3-none-any.whl
-rw-r--r--    1 root     root       5597218 Apr 10 08:50 numpy-1.20.2-cp38-cp38-linux_x86_64.whl

#11 [build 1/1] RUN pip wheel -w "/wheels" h5py=="3.2.1"
#11 sha256:52c8ae1fd876b277c3fcee034ee9d0c49c398f29167904a34f76b6047e2c1b29
#11 1.176 Looking in links: /wheels
#11 1.478 Collecting h5py==3.2.1
#11 1.694   Downloading h5py-3.2.1.tar.gz (368 kB)
#11 1.950   Installing build dependencies: started
#11 4.290   Installing build dependencies: finished with status 'done'
#11 4.291   Getting requirements to build wheel: started
#11 4.565   Getting requirements to build wheel: finished with status 'done'
#11 4.568     Preparing wheel metadata: started
#11 4.897     Preparing wheel metadata: finished with status 'done'
#11 5.393 Processing /wheels/numpy-1.20.2-cp38-cp38-linux_x86_64.whl
#11 5.393   File was already downloaded /wheels/numpy-1.20.2-cp38-cp38-linux_x86_64.whl
#11 5.403 Building wheels for collected packages: h5py
#11 5.404   Building wheel for h5py (PEP 517): started
#11 5.627   Building wheel for h5py (PEP 517): finished with status 'error'
#11 5.627   ERROR: Command errored out with exit status 1:
#11 5.627    command: /usr/bin/python3 /usr/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpfp6zj27g
#11 5.627        cwd: /tmp/pip-wheel-0ln7tye0/h5py_a41493a5e4e7463fa8aa4e1b832d6939

…

#11 5.627   Traceback (most recent call last):
#11 5.627     File "/usr/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
#11 5.627       main()
#11 5.627     File "/usr/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
#11 5.627       json_out['return_val'] = hook(**hook_input['kwargs'])
#11 5.627     File "/usr/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 204, in build_wheel
#11 5.627       return _build_backend().build_wheel(wheel_directory, config_settings,
#11 5.627     File "/tmp/pip-build-env-df_0oxkq/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 221, in build_wheel
#11 5.627       return self._build_with_temp_dir(['bdist_wheel'], '.whl',
#11 5.627     File "/tmp/pip-build-env-df_0oxkq/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 207, in _build_with_temp_dir
#11 5.627       self.run_setup()
#11 5.627     File "/tmp/pip-build-env-df_0oxkq/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 150, in run_setup
#11 5.627       exec(compile(code, __file__, 'exec'), locals())
#11 5.627     File "setup.py", line 156, in <module>
#11 5.627       setup(
#11 5.627     File "/tmp/pip-build-env-df_0oxkq/overlay/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup
#11 5.627       return distutils.core.setup(**attrs)
#11 5.627     File "/usr/lib/python3.8/distutils/core.py", line 148, in setup
#11 5.627       dist.run_commands()
#11 5.627     File "/usr/lib/python3.8/distutils/dist.py", line 966, in run_commands
#11 5.627       self.run_command(cmd)
#11 5.627     File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
#11 5.627       cmd_obj.run()
#11 5.627     File "/tmp/pip-build-env-df_0oxkq/overlay/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 299, in run
#11 5.627       self.run_command('build')
#11 5.627     File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
#11 5.627       self.distribution.run_command(command)
#11 5.627     File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
#11 5.627       cmd_obj.run()
#11 5.627     File "/usr/lib/python3.8/distutils/command/build.py", line 135, in run
#11 5.627       self.run_command(cmd_name)
#11 5.627     File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
#11 5.627       self.distribution.run_command(command)
#11 5.627     File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
#11 5.627       cmd_obj.run()
#11 5.627     File "/tmp/pip-wheel-0ln7tye0/h5py_a41493a5e4e7463fa8aa4e1b832d6939/setup_build.py", line 121, in run
#11 5.627       from Cython import __version__ as cython_version
#11 5.627   ModuleNotFoundError: No module named 'Cython'
#11 5.627   ----------------------------------------
#11 5.628   ERROR: Failed building wheel for h5py
#11 5.628 Failed to build h5py
#11 5.628 ERROR: Failed to build one or more wheels

[takluyver]

It’s definitely not an ideal solution, but what we’re doing seems to be the best available option - Python packaging isn’t really set up to understand ‘links against’ relationships or ABI compatibility.

Scipy also pins the oldest compatible numpy, and this approach is recommended by NumPy. I’m not sure why you see different behaviour for scipy - maybe because it doesn’t define a PEP 517 buildsystem, it doesn’t get build isolation by default yet. That will probably change one day, but the --no-build-isolation flag is there if you need it.

[nathanblair]

I’ll go ahead and close this as it seems like the expectation is for a separate build environment to be maintained when using pre-built wheels. And I think this case seems to be treading undocumented/undefined behavior.

I think this warrants some further research to discover the edge cases like this that fall between the cracks of using hard-coded versions for dependencies. But I leave that to the maintainers of the project to decide if that’s in their goal and scope to support.

For everyone else, there’s always requirements.txt 🤷‍♂️