Support (tempita/template) based code generation

We semi-recently started using cythonize in pandas, and I’m thinking some of the wrapping code used there might be worth pushing upstream. I’d like to get opinions before making PR(s) here.

• Integrate optional tempita rendering: https://github.com/pandas-dev/pandas/blob/master/setup.py#L97

Further ideas:

• Avoid running cythonize during python setup.py clean: https://github.com/pandas-dev/pandas/blob/master/setup.py#L463

• Enable cython coverage: https://github.com/pandas-dev/pandas/blob/master/setup.py#L442

[da-woods]

I’m basically in favour of this - I think it’s a good idea to solve this kind of problem with something that works as a separate phase (and with distinct syntax) rather than trying to load too much into Cython syntax. Designing a robust “macro-like” language seems like a fairly hard problem, especially since Cython is pretty constrained to be close to Python syntax.

I’ve generally recommended Tempita for this kind of problem (just because it’s there, and because people like Numpy and Scipy already use it), so building it into the build process a little more thoroughly seems like a good idea.

I’d prefer not killing off DEF and IF too quickly. I definitely think we should deprecate them, issue a warning, and refuse to add new features to them. But I don’t think they’re causing much harm just by existing, so I’d rather leave a good long while for people to adapt and make sure that our proposed replacement is actually fit for purpose. To me Cython 3.1 seems too soon (given that the gap between 3.0 and 3.1 is currently planned to be fairly short).

[jakirkham]

Yeah some kind of templating in Cython could be pretty powerful, but probably needs to be done thoughtfully.

Tempita does seem like an easy way forward. Think there are some libraries already using it (like scikit-learn). There are other templating libraries like this in the wild (like Jinja) that might also make sense. It might be worth considering how to templating while maintaining Python looking code (valuable for linting, reading by new devs, pure Python mode, etc.).

Fused types can already be pretty useful on their own. One can even use these to do SFINAE like things in Cython. For example here’s an enable_if analogue in Cython ( https://github.com/cython/cython/issues/2660#issuecomment-687493653 ).

Have wondered if C++ style templating could be used in Cython directly.

Maybe it is worth enumerating what use cases templating would support?