Cython does not respect dependency for generating `.pyx.c` file

Describe the bug

Building a Python extension from a generated Cython file (_ufuncs.pyx) fails with:

[1/539] Compiling Cython source scipy/special/_ufuncs.pyx
FAILED: scipy/special/_ufuncs.cpython-39-x86_64-linux-gnu.so.p/scipy/special/_ufuncs.pyx.c 
cython --fast-fail -3 -o scipy/special/_ufuncs.cpython-39-x86_64-linux-gnu.so.p/scipy/special/_ufuncs.pyx.c scipy/special/_ufuncs.pyx

Error compiling Cython file:
------------------------------------------------------------
...
# This file is automatically generated by _generate_pyx.py.
# Do not edit manually!

include "_ufuncs_extra_code_common.pxi"
^
------------------------------------------------------------

The relevant meson.build content is:

# Generates the `_ufuncs.pyx` file that fails to build later
_generate_cy = custom_target('_generate_cy',
  output : ['_ufuncs.pyx',
            '_ufuncs_defs.h',
            '_ufuncs_cxx.pyx',
            '_ufuncs_cxx.pxd',
            '_ufuncs_cxx_defs.h',
            '_ufuncs.pyi',
            'cython_special.pyx',
            'cython_special.pxd'
           ],
  input : ['_generate_pyx.py', 'functions.json', 'add_newdocs.py'],
  command : [py3, '@INPUT0@', '-o', '@OUTDIR@'],
)

# Simply copy over the files we need to the build dir
_ufuncs_pxi_pxd_sources = custom_target('_ufuncs_pxi_pxd',
  output : [
    '__init__.py',
    ...
    '_ufuncs_extra_code.pxi',
    '_ufuncs_extra_code_common.pxi',
  ],
  input : [
    '__init__.py',
    ...
    '_ufuncs_extra_code.pxi',
    '_ufuncs_extra_code_common.pxi',
  ],
  command : ['cp', '@INPUT@', '-t', '@OUTDIR@'],
)

# FIXME: the generated ninja.build file is missing the dependency between
#       copying these files into the build dir and the `.pyx -> .c` stage.
#       For the `.c -> .so` stage the dependency (see `||` in the `build.ninja`
#       file) is picked up correctly. This only shows up when running the build
#       with `-j 1` (when the steps are run in parallel, the copying of sources
#       seems to always happen quickly enough. This will likely only be fixed
#       by the "structured sources" PR on the Meson repo.
_ufuncs_pxi_pxd_dep = declare_dependency(sources: _ufuncs_pxi_pxd_sources)

py3.extension_module('_ufuncs',
  [ufuncs_sources, _generate_cy[0], _dummy_init],
  c_args : [numpy_nodepr_api, use_math_defines],
  include_directories: [inc_np, '../_lib', '../_build_utils/src'],
  dependencies : [py3_dep, lapack, npymath_lib, _ufuncs_pxi_pxd_dep],  # We have the right dependency here
  link_with: [
    amos_lib,
    cdflib_lib,
    cephes_lib,
    mach_lib,
    specfun_lib,
  ],
  install : true,
  subdir : 'scipy/special'
)

What seems to be happening is that a ninja order dependency goes missing. These are the build rules generated:

build scipy/special/_ufuncs.pyx scipy/special/_ufuncs_defs.h scipy/special/_ufuncs_cxx.pyx scipy/special/_ufuncs_cxx.pxd scipy/special/_ufuncs_cxx_defs.h scipy/special/_ufuncs.pyi scipy/special/cython_special.pyx scipy/special/cython_special.pxd: CUSTOM_COMMAND ../scipy/special/_generate_pyx.py ../scipy/special/functions.json ../scipy/special/add_newdocs.py | /home/rgommers/anaconda3/envs/scipy-meson/bin/python3.9
 COMMAND = /home/rgommers/anaconda3/envs/scipy-meson/bin/python3.9 ../scipy/special/_generate_pyx.py -o scipy/special
 description = Generating$ _generate_cy$ with$ a$ custom$ command

build scipy/special/__init__.py scipy/special/_agm.pxd scipy/special/_boxcox.pxd scipy/special/_cephes.pxd scipy/special/_complexstuff.pxd scipy/special/_convex_analysis.pxd scipy/special/_cunity.pxd scipy/special/_digamma.pxd scipy/special/_ellip_harm.pxd scipy/special/_ellip_harm_2.pxd scipy/special/_ellipk.pxd scipy/special/_evalpoly.pxd scipy/special/_exprel.pxd scipy/special/_factorial.pxd scipy/special/_hyp0f1.pxd scipy/special/_hypergeometric.pxd scipy/special/_lambertw.pxd scipy/special/_legacy.pxd scipy/special/_loggamma.pxd scipy/special/_ndtri_exp.pxd scipy/special/_sici.pxd scipy/special/_spence.pxd scipy/special/_spherical_bessel.pxd scipy/special/_trig.pxd scipy/special/_wright_bessel.pxd scipy/special/_xlogy.pxd scipy/special/orthogonal_eval.pxd scipy/special/sf_error.pxd scipy/special/sph_harm.pxd scipy/special/_cython_special.pxi scipy/special/_cython_special_custom.pxi scipy/special/_ufuncs_extra_code.pxi scipy/special/_ufuncs_extra_code_common.pxi: CUSTOM_COMMAND ../scipy/special/__init__.py ../scipy/special/_agm.pxd ../scipy/special/_boxcox.pxd ../scipy/special/_cephes.pxd ../scipy/special/_complexstuff.pxd ../scipy/special/_convex_analysis.pxd ../scipy/special/_cunity.pxd ../scipy/special/_digamma.pxd ../scipy/special/_ellip_harm.pxd ../scipy/special/_ellip_harm_2.pxd ../scipy/special/_ellipk.pxd ../scipy/special/_evalpoly.pxd ../scipy/special/_exprel.pxd ../scipy/special/_factorial.pxd ../scipy/special/_hyp0f1.pxd ../scipy/special/_hypergeometric.pxd ../scipy/special/_lambertw.pxd ../scipy/special/_legacy.pxd ../scipy/special/_loggamma.pxd ../scipy/special/_ndtri_exp.pxd ../scipy/special/_sici.pxd ../scipy/special/_spence.pxd ../scipy/special/_spherical_bessel.pxd ../scipy/special/_trig.pxd ../scipy/special/_wright_bessel.pxd ../scipy/special/_xlogy.pxd ../scipy/special/orthogonal_eval.pxd ../scipy/special/sf_error.pxd ../scipy/special/sph_harm.pxd ../scipy/special/_cython_special.pxi ../scipy/special/_cython_special_custom.pxi ../scipy/special/_ufuncs_extra_code.pxi ../scipy/special/_ufuncs_extra_code_common.pxi | /usr/bin/cp
 COMMAND = /usr/bin/cp ../scipy/special/__init__.py ../scipy/special/_agm.pxd ../scipy/special/_boxcox.pxd ../scipy/special/_cephes.pxd ../scipy/special/_complexstuff.pxd ../scipy/special/_convex_analysis.pxd ../scipy/special/_cunity.pxd ../scipy/special/_digamma.pxd ../scipy/special/_ellip_harm.pxd ../scipy/special/_ellip_harm_2.pxd ../scipy/special/_ellipk.pxd ../scipy/special/_evalpoly.pxd ../scipy/special/_exprel.pxd ../scipy/special/_factorial.pxd ../scipy/special/_hyp0f1.pxd ../scipy/special/_hypergeometric.pxd ../scipy/special/_lambertw.pxd ../scipy/special/_legacy.pxd ../scipy/special/_loggamma.pxd ../scipy/special/_ndtri_exp.pxd ../scipy/special/_sici.pxd ../scipy/special/_spence.pxd ../scipy/special/_spherical_bessel.pxd ../scipy/special/_trig.pxd ../scipy/special/_wright_bessel.pxd ../scipy/special/_xlogy.pxd ../scipy/special/orthogonal_eval.pxd ../scipy/special/sf_error.pxd ../scipy/special/sph_harm.pxd ../scipy/special/_cython_special.pxi ../scipy/special/_cython_special_custom.pxi ../scipy/special/_ufuncs_extra_code.pxi ../scipy/special/_ufuncs_extra_code_common.pxi -t scipy/special
 description = Generating$ _ufuncs_pxi_pxd$ with$ a$ custom$ command


build scipy/special/_ufuncs.cpython-39-x86_64-linux-gnu.so.p/scipy/special/_ufuncs.pyx.c: cython_COMPILER scipy/special/_ufuncs.pyx
 ARGS = --fast-fail -3 -o scipy/special/_ufuncs.cpython-39-x86_64-linux-gnu.so.p/scipy/special/_ufuncs.pyx.c

build scipy/special/_ufuncs.cpython-39-x86_64-linux-gnu.so.p/meson-generated_scipy_special__ufuncs.pyx.c.o: c_COMPILER scipy/special/_ufuncs.cpython-39-x86_64-linux-gnu.so.p/scipy/special/_ufuncs.pyx.c || scipy/__init__.py scipy/special/__init__.py scipy/special/_agm.pxd scipy/special/_boxcox.pxd scipy/special/_cephes.pxd scipy/special/_complexstuff.pxd scipy/special/_convex_analysis.pxd scipy/special/_cunity.pxd scipy/special/_cython_special.pxi scipy/special/_cython_special_custom.pxi scipy/special/_digamma.pxd scipy/special/_ellip_harm.pxd scipy/special/_ellip_harm_2.pxd scipy/special/_ellipk.pxd scipy/special/_evalpoly.pxd scipy/special/_exprel.pxd scipy/special/_factorial.pxd scipy/special/_hyp0f1.pxd scipy/special/_hypergeometric.pxd scipy/special/_lambertw.pxd scipy/special/_legacy.pxd scipy/special/_loggamma.pxd scipy/special/_ndtri_exp.pxd scipy/special/_sici.pxd scipy/special/_spence.pxd scipy/special/_spherical_bessel.pxd scipy/special/_trig.pxd scipy/special/_ufuncs_extra_code.pxi scipy/special/_ufuncs_extra_code_common.pxi scipy/special/_wright_bessel.pxd scipy/special/_xlogy.pxd scipy/special/orthogonal_eval.pxd scipy/special/sf_error.pxd scipy/special/sph_harm.pxd
 DEPFILE = scipy/special/_ufuncs.cpython-39-x86_64-linux-gnu.so.p/meson-generated_scipy_special__ufuncs.pyx.c.o.d
 DEPFILE_UNQUOTED = scipy/special/_ufuncs.cpython-39-x86_64-linux-gnu.so.p/meson-generated_scipy_special__ufuncs.pyx.c.o.d
 ARGS = -Iscipy/special/_ufuncs.cpython-39-x86_64-linux-gnu.so.p -Iscipy/special -I../scipy/special -I/home/rgommers/anaconda3/envs/scipy-meson/lib/python3.9/site-packages/numpy/core/include -Iscipy/_lib -I../scipy/_lib -I../scipy/_build_utils/src -Iscipy -I/home/rgommers/anaconda3/envs/scipy-meson/include/python3.9 -I/home/rgommers/anaconda3/envs/scipy-meson/include -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -w -std=c99 -O2 -g -Wno-unused-function -Wno-conversion -Wno-unused-but-set-variable -Wno-misleading-indentation -Wno-incompatible-pointer-types -fPIC -DNPY_NO_DEPRECATED_API=NPY_1_9_API_VERSION

If I inspect the files in the build dir, the .pxi files are indeed missing:

$ ls build/scipy/special
_comb.cpython-39-x86_64-linux-gnu.so.p           _precompute
cython_special.cpython-39-x86_64-linux-gnu.so.p  specfun.cpython-39-x86_64-linux-gnu.so
cython_special.pxd                               specfun.cpython-39-x86_64-linux-gnu.so.p
cython_special.pyx                               specfunmodule.c
_ellip_harm_2.cpython-39-x86_64-linux-gnu.so.p   _test_round.cpython-39-x86_64-linux-gnu.so.p
libamos.a                                        tests
libamos.a.p                                      _ufuncs.cpython-39-x86_64-linux-gnu.so.p
libcdflib.a                                      _ufuncs_cxx.cpython-39-x86_64-linux-gnu.so.p
libcdflib.a.p                                    _ufuncs_cxx_defs.h
libcephes.a                                      _ufuncs_cxx.pxd
libcephes.a.p                                    _ufuncs_cxx.pyx
libmach.a                                        _ufuncs_defs.h
libmach.a.p                                      _ufuncs.pyi
libspecfun.a                                     _ufuncs.pyx
libspecfun.a.p

If I understand what Ninja is doing correctly, this line:

build scipy/special/_ufuncs.cpython-39-x86_64-linux-gnu.so.p/scipy/special/_ufuncs.pyx.c: cython_COMPILER scipy/special/_ufuncs.pyx

is missing the || scipy/__init__.py .... scipy/special/_ufuncs_extra_code_common.pxi ... order dependency that the next line does have.

To Reproduce

I can write a standalone test case, but given that it’s a bit of work and I’m not 100% sure what the best way is to write a test for such a timing sensitive order dependency issue, I wanted to describe my actual failure first. Should the test case use -j 1, somehow inspect the generated ninja build rules, or something else?

In the mean time, this is reproducible from https://github.com/rgommers/scipy/tree/meson-issue-ufuncs-pyx with:

conda env create -f environment.yml
conda activate scipy-dev
python -m pip install git+https://github.com/mesonbuild/meson.git@master

meson setup build --prefix=$PWD/installdir
ninja -C build -j 1  # it's timing-dependent, so build with 1 job to avoid that

Expected behavior Build should respect order dependency, and hence Cython should not fail on a missing .pxi file that is a dependency.

system parameters

• native build
• operating system: Arch Linux
• Python version: 3.9.2
• meson --version: 0.58.999
• ninja --version: 1.10.2

@dcbaker I know gh-8775 is supposed to address this in the long term, but I believe there will still be cases where we will need custom_target in combination with compiling Cython code - we have a lot of different cases where we codegen .pyx or .pxd files. ~EDIT: we also need it for adding --cplus for example.~

[eli-schwartz]

I agree with that analysis… a test case would involve making a minimal example that exercises the need for transpile dependencies, and then having the test case attempt to build ninja scipy/special/_ufuncs.cpython-39-x86_64-linux-gnu.so.p/scipy/special/_ufuncs.pyx.c rather than attempting to hit the problem via timing.

The alternatives to transpile_depend_files would be:

• a generic buildorder function for binding new order-only dependencies, as @Jehan asked for in #8123
• adding a dyndep rule that somehow runs the transpiler in “create depfile but don’t build” mode rather than the usual “create depfile as a secondary artifact while building”. This could be viewed, in some ways, as bringing us back to the dark ages of https://en.wikipedia.org/wiki/Makedepend

[eli-schwartz]

I don’t think that will work… the .pxi, creation rule, the .c creation rule, and the .o compiler rule all end up in the build graph either way, queued for building, and we need a relationship rule for .pxi -> .c, the same way we have a .c -> .o relationship rule.

We can do that with the generated headers approach (add .pxi output files as additional sources), we just need a way to “push” that from the “before .o” to “before .c” stage.