BUG: Cython 0.29.31 release breaks building SciPy from source

Describe your issue.

This comes from failures seen in SymPy CI: https://github.com/sympy/sympy/pull/23847

The release of Cython 0.29.31 means that building SciPy from source with pip fails both with the latest release SciPy 1.8.1 and also the main branch.

It looks like pyproject.toml on the 1.8.x branch needs to have a better upper bound for the Cython version since it only requires <3.0 which does not prevent new releases in Cython’s 0.29.x series from breaking the build: https://github.com/scipy/scipy/blob/b76137f0d4fd82bd74b2e51af33af3ece44e45bc/pyproject.toml#L1-L14

Reproducing Code Example

$ pip install -U pip setuptools wheel cython numpy
Collecting pip
  Downloading pip-22.2.1-py3-none-any.whl (2.0 MB)
     |████████████████████████████████| 2.0 MB 4.8 MB/s 
Collecting setuptools
  Downloading setuptools-63.2.0-py3-none-any.whl (1.2 MB)
     |████████████████████████████████| 1.2 MB 28.2 MB/s 
Collecting wheel
  Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
Collecting cython
  Downloading Cython-0.29.31-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (1.9 MB)
     |████████████████████████████████| 1.9 MB 2.8 MB/s 
Collecting numpy
  Using cached numpy-1.23.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.1 MB)
Installing collected packages: pip, setuptools, wheel, cython, numpy
  Attempting uninstall: pip
    Found existing installation: pip 20.2.3
    Uninstalling pip-20.2.3:
      Successfully uninstalled pip-20.2.3
  Attempting uninstall: setuptools
    Found existing installation: setuptools 49.2.1
    Uninstalling setuptools-49.2.1:
      Successfully uninstalled setuptools-49.2.1
Successfully installed cython-0.29.31 numpy-1.23.1 pip-22.2.1 setuptools-63.2.0 wheel-0.37.1
$ pip install --no-binary scipy scipy
Collecting scipy
  Downloading scipy-1.8.1.tar.gz (38.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 38.2/38.2 MB 8.0 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

Error message

$ pip install --no-binary scipy scipy
Collecting scipy
  Downloading scipy-1.8.1.tar.gz (38.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 38.2/38.2 MB 8.0 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [102 lines of output]
      Running scipy/special/_generate_pyx.py
      Running scipy/stats/_generate_pyx.py
      Running scipy/linalg/_generate_pyx.py
      Processing scipy/special/_ufuncs.pyx
      Processing scipy/special/_test_round.pyx
      Processing scipy/special/_ufuncs_cxx.pyx
      Processing scipy/special/_ellip_harm_2.pyx
      Processing scipy/special/cython_special.pyx
      Processing scipy/special/_comb.pyx
      Processing scipy/spatial/_voronoi.pyx
      Processing scipy/spatial/_hausdorff.pyx
      Processing scipy/spatial/_qhull.pyx
      Processing scipy/spatial/_ckdtree.pyx
      Processing scipy/spatial/transform/_rotation.pyx
      Processing scipy/interpolate/_ppoly.pyx
      Processing scipy/interpolate/_bspl.pyx
      Processing scipy/interpolate/interpnd.pyx
      Processing scipy/cluster/_vq.pyx
      Processing scipy/cluster/_hierarchy.pyx
      Processing scipy/cluster/_optimal_leaf_ordering.pyx
      Processing scipy/_lib/messagestream.pyx
      Processing scipy/_lib/_test_deprecation_call.pyx
      Processing scipy/_lib/_test_deprecation_def.pyx
      Processing scipy/_lib/_ccallback_c.pyx
      Processing scipy/signal/_upfirdn_apply.pyx
      Processing scipy/signal/_max_len_seq_inner.pyx
      Processing scipy/signal/_sosfilt.pyx
      Processing scipy/signal/_spectral.pyx
      Processing scipy/signal/_peak_finding_utils.pyx
      Processing scipy/io/matlab/_streams.pyx
      Processing scipy/io/matlab/_mio5_utils.pyx
      Processing scipy/io/matlab/_mio_utils.pyx
      Processing scipy/stats/_stats.pyx
      Processing scipy/stats/_qmc_cy.pyx
      Processing scipy/stats/_biasedurn.pyx
      Processing scipy/stats/_sobol.pyx
      Processing scipy/stats/_boost/src/nbinom_ufunc.pyx
      Processing scipy/stats/_boost/src/beta_ufunc.pyx
      
      Error compiling Cython file:
      ------------------------------------------------------------
      ...
          return np.array(result, dtype=np.int64)
      
      
      @cython.wraparound(False)
      @cython.boundscheck(False)
      def _weightedrankedtau(ordered[:] x, ordered[:] y, intp_t[:] rank, weigher, bool additive):
      ^
      ------------------------------------------------------------
      
      _stats.pyx:173:0: Referring to a memoryview typed argument directly in a nested closure function is not supported in Cython 0.x. Either upgrade to Cython 3, or assign the argument to a local variable and use that in the nested function.
      Processing scipy/stats/_boost/src/binom_ufunc.pyx
      Traceback (most recent call last):
        File "/tmp/pip-install-wnazhy0r/scipy_3ee8a876e4f54aa7bfa3d9fdba592754/tools/cythonize.py", line 317, in <module>
          main()
        File "/tmp/pip-install-wnazhy0r/scipy_3ee8a876e4f54aa7bfa3d9fdba592754/tools/cythonize.py", line 313, in main
          find_process_files(root_dir)
        File "/tmp/pip-install-wnazhy0r/scipy_3ee8a876e4f54aa7bfa3d9fdba592754/tools/cythonize.py", line 302, in find_process_files
          for result in pool.imap_unordered(lambda args: process(*args), jobs):
        File "/usr/lib/python3.8/multiprocessing/pool.py", line 868, in next
          raise value
        File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
          result = (True, func(*args, **kwds))
        File "/tmp/pip-install-wnazhy0r/scipy_3ee8a876e4f54aa7bfa3d9fdba592754/tools/cythonize.py", line 302, in <lambda>
          for result in pool.imap_unordered(lambda args: process(*args), jobs):
        File "/tmp/pip-install-wnazhy0r/scipy_3ee8a876e4f54aa7bfa3d9fdba592754/tools/cythonize.py", line 236, in process
          processor_function(fromfile, tofile, cwd=path)
        File "/tmp/pip-install-wnazhy0r/scipy_3ee8a876e4f54aa7bfa3d9fdba592754/tools/cythonize.py", line 102, in process_pyx
          raise Exception('Cython failed')
      Exception: Cython failed
      setup.py:486: UserWarning: Unrecognized setuptools command ('dist_info --egg-base /tmp/pip-modern-metadata-tcs7khbn'), proceeding with generating Cython sources and expanding templates
        warnings.warn("Unrecognized setuptools command ('{}'), proceeding with "
      Running from SciPy source directory.
      Cythonizing sources
      Traceback (most recent call last):
        File "setup.py", line 350, in generate_cython
          import pip
      ModuleNotFoundError: No module named 'pip'
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "/home/oscar/current/sympy/sympy.git/tmpvenv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/home/oscar/current/sympy/sympy.git/tmpvenv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/home/oscar/current/sympy/sympy.git/tmpvenv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 164, in prepare_metadata_for_build_wheel
          return hook(metadata_directory, config_settings)
        File "/tmp/pip-build-env-nvuzm8iv/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 174, in prepare_metadata_for_build_wheel
          self.run_setup()
        File "/tmp/pip-build-env-nvuzm8iv/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 267, in run_setup
          super(_BuildMetaLegacyBackend,
        File "/tmp/pip-build-env-nvuzm8iv/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 158, in run_setup
          exec(compile(code, __file__, 'exec'), locals())
        File "setup.py", line 628, in <module>
          setup_package()
        File "setup.py", line 612, in setup_package
          generate_cython()
        File "setup.py", line 352, in generate_cython
          raise RuntimeError("Running cythonize failed!")
      RuntimeError: Running cythonize failed!
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

SciPy/NumPy/Python version information

SciPy 1.8.1 and current main, numpy 1.23.1, Any version of Python or PyPy (I think), Cython 0.29.31

[mattip]

I think this is a duplicate of #16718 and fixed in #16719

[rgommers]

This is fixed on both ends of the line now, so can be closed.