Source build can only find 'system' installation of Cython, not --user or venv

• Operating System: Ubuntu 20.04 on aarch64 (Amazon c6g instances)
• Python version 3.8, installed via apt-get install python-3.8
• h5py version 3.1.0
• HDF5 version 1.10.0 (from apt)
• The full traceback/stack trace shown (if it appears)

I’m building h5py from source using

pip3 install numpy cython pkgconfig H5PY_SETUP_REQUIRES=0 pip3 install h5py

to avoid the error seen in #1760 from building against old versions of numpy.

Depending on user and conditions under which this is run, I get different results. When run naively as root, resulting in installation in /usr/local, the build appears to succeed.

When run (with PATH set appropriately to include the cython installation)

• as a non-root user, or
• as root with the --user argument passed to the pip commands, or
• as root in an active venv

I get the following error:

  running build_ext
  Traceback (most recent call last):
    File "/tmp/tmpkxnnzk0s", line 280, in <module>
      main()
    File "/tmp/tmpkxnnzk0s", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/tmp/tmpkxnnzk0s", line 204, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
    File "/tmp/pip-build-env-msqi99rv/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 216, in build_wheel
      return self._build_with_temp_dir(['bdist_wheel'], '.whl',
    File "/tmp/pip-build-env-msqi99rv/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 202, in _build_with_temp_dir
      self.run_setup()
    File "/tmp/pip-build-env-msqi99rv/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 145, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 158, in <module>
      setup(
    File "/tmp/pip-build-env-msqi99rv/overlay/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup
      return distutils.core.setup(**attrs)
    File "/usr/lib/python3.8/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/lib/python3.8/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-msqi99rv/overlay/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 290, in run
      self.run_command('build')
    File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/usr/lib/python3.8/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/tmp/pip-install-4cfsrgha/h5py/setup_build.py", line 121, in run
      from Cython import __version__ as cython_version
  ModuleNotFoundError: No module named 'Cython'
  ----------------------------------------
  ERROR: Failed building wheel for h5py

It seems that there’s something in how the wheel build works that relies on finding a system-wide installation of Cython, and not a private installation.

[takluyver]

If you set H5PY_SETUP_REQUIRES=0, you probably also need to use pip install --no-build-isolation.

With recent changes in the packaging ecosystem (see PEPs 517 & 518 for details), pip now creates an isolated temporary environment when it needs to build a package from source. Inside that environment, it installs whatever the package says it needs to build, and then runs the build. The idea is that a package can be built more consistently if it’s not affected by the packages in your working environment. h5py normally says it requires numpy & Cython, but setting H5PY_SETUP_REQUIRES=0 disables that, so pip tries to build it without them.

The --no-build-isolation flag for pip install tells pip that you have already installed the build dependencies and want to build with whatever packages are in the environment where pip itself is installed.

As far as I know, it doesn’t treat your system installation any differently for this. I’d guess that when you installed as root, it used an older version of pip which didn’t do build isolation.

[PhilMiller]

In my case, I think that if SETUP_REQUIRES had specified the packages, but not a version, it would have worked just fine. I was using the latest versions, because they have portability fixes that I need (see: aarch64), and already had cached wheels for them as a result of earlier installation in the script, which the build would have picked up and installed. So, we can’t exactly detect installed versions, but it could be easier to ask for “don’t back-date to the oldest ABI”