Extending with Pythran - next steps

After the 1.6.x branch we merged gh-8306, which adds Pythran as an optional build dependency. This should get us some more experience with how much of an improvement over Cython this is, and if there are any pain points or showstoppers to keep Pythran as a build-time dependency for the next release.

Things that are necessary before the next release (from https://github.com/scipy/scipy/pull/8306#issuecomment-751397248):

• We don’t have PyPy and AIX CI, so that isn’t tested yet - ensure that all works fine. PyPy in particular.
• CI is using clang-cl on Windows - figure out what that means for wheel and conda builds, and other packaging.
• Pythran is packaged for conda-forge but not defaults, so that may give some delays for introducing the next release (we need to give the Anaconda team a heads up, at the latest around the 1.7.x branch point)
• check SIMD instructions included by default, and see if we can use more of them for improved performance. need to ensure it’s SSE3 in wheels I believe.
• Also document how to enable all SIMD instructions detected for local builds.
• Update the build instructions and release notes once we decide to keep Pythran enabled for 1.7.0
• Update the scipy-wheels repo

Pythran usage related to convince ourselves that it’s a good addition, and preferable to Cython in some circumstances:

• try on a more complex Cython extension, such as ndimage/_ni_label.pyx (which uses fused types)
• try on suitable pure Python code (e.g. integrate.quad_vec, stats.ks_2samp)
• add a good set of benchmarks
• try the Pythran OpenMP support as well - we won’t want that in binaries, but for people who want optimal performance and are happy to build from source, getting it for free may be useful.
• ~Update the way PythranExtension is used in signal/setup.py after updates in Pythran itself. see gh-8306 for discussion.~ EDIT: this was about needing to use full paths (e.g., sources=["scipy/signal/_max_len_seq_inner.py"]) because PythranExtension doesn’t handle subpackages. I don’t care that much though, if we plan to get rid of distutils anyway, why bother putting work into this.

Cc @serge-sans-paille

[rgommers]

By the way, SciPy 1.7.0 has been out for 5 days and no one has reported a Pythran related build failure yet - that’s a good sign:)

[rgommers]

I’ve not tried copying wheels, it’ll be a Linux wheel if anything.

That’s probably worth trying then. Nothing should change for you if you can use wheels or conda packages. That’s the whole reason why we can be way more liberal with build dependencies vs. runtime dependencies - it’s O(100-1000) people who should need the former, vs. O(10,000,000) who need the latter. We even have nightlies if you need something that’s not in the latest release: https://anaconda.org/scipy-wheels-nightly/scipy/files.

One key win would be to use a dependency resolver tool that would tell me all the packages needed for a given environment/install, and would then go out and retrieve the source tarball, or wheels, for a specific platform (or all platforms).

You should be able to do this with Conda, or with Poetry as well if you must have wheels I think. E.g. try https://conda.github.io/conda-pack/ to move a complete environment at once.

I have only limited experience with clusters, but I did find using conda to install everything was a lot nicer than trying to interact with the (usually outdated) Python version installed by the admins.