Treating Cython like a language?

Context: I’m working on building SciPy with Meson. SciPy is a mixed-language library (Python, Cython, C, C++, Fortran), for more details of Meson + SciPy see https://github.com/scipy/scipy/issues/13615.

I’d like to get your feedback on a design question on how to integrate tools like Cython, but later perhaps also f2py and Pythran. I’m combining all of these because they interact with a build system in the same way:

• They all generate C or C++ code.
• The can be invoked as a command-line tool to generate only C/C++ (e.g., cython somefile.pyx)
• They can be used like a compiler to generate a Python extension module (e.g. cythonize -a -i somefile.pyx).
• They have build system integration with numpy.distutils and setuptools, so you can do something like setup(ext_modules = cythonize("somefile.pyx")) in a setup.py file to obtain a Python extension module.

None of the three is supported in Meson directly; Cython is by far the most widely used one and is tested in Meson’s test cases (e.g, from here):

pyx_c = custom_target('storer_pyx',
  output : 'storer_pyx.c',
  input : 'storer.pyx',
  command : [cython, '@INPUT@', '-o', '@OUTPUT@'],
)

slib = py3.extension_module('storer',
  'storer.c', pyx_c,
  dependencies : py3_dep)

pydir = meson.current_build_dir()

The custom_target and then py3.extension_module approach should work for all these tools I think, but it’s going to be very verbose. It takes 9 lines per extension module in meson.build compared to 1-2 lines with the current setup.py approach. Some of this ('@INPUT@', '-o', '@OUTPUT@', dependencies : py3_dep) is only boiler-plate.

There are compiler directives (e.g. --fast-fail, --language-level=3, --cplus, for Cython; -fopenmp, -DUSE_XSIMD for Pythran) that one may want to set by default or switch around easily, as well as include file types (.pxi) and a templating format (.pyx.in) that may make the difference in verbosity even larger. In SciPy right now, the approach is to use a separate pre-build script (tools/cythonize.py) to do that for all Cython sources. That seems like a bad idea (e.g., it needs to maintain its own ad-hoc build cache).

Therefore, given that Cython is an actual language that just happens to transpile to C (or, optionally, C++), and has a relatively large user base, I wonder if it makes sense to treat it like a supported language in Meson rather than as an external tool one must use custom_target for? So one could just write:

# cython is now treated like a compiler, so `cython = find_program('cython')`
# is also not needed.
project('projname', 'cython')

slib = py3.extension_module('storer',
  'storer.pyx',
  dependencies : py3_dep)  # this line also seems like boiler plate, it's always needed to build a Python extension module?

I’m wondering if this makes sense to support, and if not if there’s anything else you can recommend to make creating Python modules from Cython more ergonomic?

[dcbaker]

Right, I have the same problem with some Rust code, actually. (in my case I’m using bindgen to create a wrapper around a C library, but it’s the same problem)

I’m currently writing the structured input idea. See if it works out.

[dcbaker]

ATM anything generated is going to be silently ignored. I think I need to sit down and write my structured input proposal to make that work, which I need to do for Rust anyway.