Tracking issue for cross-compilation needs and issues

Cross-compiling with numpy.distutils isn’t really supported, and we regularly receive issues related to that. With the move to Meson (gh-13615), cross-compiling becomes supported. This issue aims to provide an overview of cross-compiling needs users have expressed in the past. That will help when implementing cross-compilation support and coming up with relevant test cases.

• Yocto Linux has come up multiple times:
  • gh-8571 (main issue, still open - mentions successful result in this comment on gh-8571)
  • gh-10770
  • gh-8226
• Buildroot has come up several times as well:
  • gh-9875
  • https://news.ycombinator.com/item?id=27953673 (multiple mentions of cross-compiling SciPy in those comments)
• Pyodide is cross-compiling to WebAssembly:
  • gh-15290 (overview issue)
  • https://github.com/scipy/scipy/pull/15288#issuecomment-1002834295 (some discussion on Meson & Emscripten)
• Void Linux is cross-compiling SciPy; the numpy.distutils based build worked, and the move to Meson yielded one problem - see gh-16783. For the cross-compile support files in Void, see here
• Linaro has a repo for cross-compiling packages for Windows on ARM at https://gitlab.com/Linaro/windowsonarm/packages. It does not contain SciPy yet; it’d be interesting to see that though. Meson cross-compiling support should work with that toolchain; invoking the build through pip or build may have a few rough edges (not sure until it’s tried). See this NumPy mailing list thread for more details.

Both Yocto and Buildroot are ways to generate an embedded Linux distribution through cross-compilation. We should aim to test at least one of those, preferably in CI. Related build systems that were mentioned are OpenEmbedded and Bitbake. https://github.com/scipy/scipy/issues/8571#issuecomment-715877299 mentions crossenv (not quite a build system, more virtualenv trickery for distutils) to build for Yocto.

• Cross-compiling to 32-bit ARM: gh-10727
• Cross-compiling on Cray XT (old, not so relevant anymore): gh-1857
• An effort to make numpy.distutils more amenable to cross-compilation: https://github.com/numpy/numpy/issues/17620 (also mentions Yocto/OpenEmbedded).
• The one place we are successfully cross-compiling is in https://github.com/MacPython/scipy-wheels to produce arm64 macOS wheels from x86_64 macOS CI jobs - and that happens to work because the differences between x86_64 and arm64 builds are fairly limited.

[gwbres]

@rgommers congratulations, it works. I manually modified the “libnpyrandom” and libnpymath" path, in scipy/meson.build, and it worked.

what is the best way to apply a patch

If you have a patch file (*.patch) that was generated so that it applies correctly at the base of the scipy repo, you can simply put it in buildroot/package/python-scipy, and buildroot will automatically apply it before “configure” step, see other existing patches

ls buildroot/package/*/*.patch

[rgommers]

Thanks @gwbres, that worked out of the box on Arch Linux after installing cpio and upgrading to GCC 9.

support/misc/cross-compilation.conf.in generates for the scipy package build this cross file (as ./output/build/python-scipy-1.9.0/build/cross-compilation.conf):

# Note: Buildroot's and Meson's terminologies differ about the meaning
# of 'build', 'host' and 'target':
# - Buildroot's 'host' is Meson's 'build'
# - Buildroot's 'target' is Meson's 'host'

[binaries]
c = '/home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/bin/aarch64-buildroot-linux-gnu-gcc'
cpp = '/home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/bin/aarch64-buildroot-linux-gnu-g++'
ar = '/home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/bin/aarch64-buildroot-linux-gnu-gcc-ar'
strip = '/home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/bin/aarch64-buildroot-linux-gnu-strip'
cmake = '/usr/bin/cmake'
fortran = '/home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/bin/aarch64-buildroot-linux-gnu-gfortran'
pkgconfig = '/home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/bin/pkgconf'
g-ir-compiler = '/home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/aarch64-buildroot-linux-gnu/sysroot/usr/bin/g-ir-compiler'
g-ir-scanner = '/home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/aarch64-buildroot-linux-gnu/sysroot/usr/bin/g-ir-scanner'

[built-in options]
c_args = ['-D_LARGEFILE_SOURCE', '-D_LARGEFILE64_SOURCE', '-D_FILE_OFFSET_BITS=64', '-Os', '-g0', '-D_FORTIFY_SOURCE=1']
c_link_args = []
cpp_args = ['-D_LARGEFILE_SOURCE', '-D_LARGEFILE64_SOURCE', '-D_FILE_OFFSET_BITS=64', '-Os', '-g0', '-D_FORTIFY_SOURCE=1']
cpp_link_args = []
fortran_args = ['-D_LARGEFILE_SOURCE', '-D_LARGEFILE64_SOURCE', '-D_FILE_OFFSET_BITS=64', '-Os', '-g0', '-D_FORTIFY_SOURCE=1']
fortran_link_args = []
wrap_mode = 'nodownload'
cmake_prefix_path = '/home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/aarch64-buildroot-linux-gnu/sysroot/usr/lib/cmake'

[properties]
needs_exe_wrapper = true
sys_root = '/home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/aarch64-buildroot-linux-gnu/sysroot'
pkg_config_libdir = '/home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/aarch64-buildroot-linux-gnu/sysroot/usr/lib/pkgconfig:/home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/aarch64-buildroot-linux-gnu/sysroot/usr/share/pkgconfig'
pkg_config_static = 'false'
# enable meson build to pass a toolchain file to cmake
cmake_toolchain_file = '/home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/share/buildroot/toolchainfile.cmake'
cmake_defaults = false

[host_machine]
system = 'linux'
cpu_family = 'aarch64'
cpu = 'cortex-a72'
endian = 'little'

Looking at the build log in output/build/python-scipy-1.9.0/build/meson-logs/meson-log.txt, it finds the host Python:

Program python3 found: YES (/home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/bin/python3)

Then here is when it goes wrong in determining the correct path to libnpymath.a:

Searching for 'python-3.10' via pkgconfig lookup in LIBPC
Pkg-config binary for 1 is not cached.
Pkg-config binary for 1 specified from cross file, native file, or env var as ['/home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/bin/pkgconf']
Found pkg-config: /home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/bin/pkgconf (1.6.3)
Determining dependency 'python-3.10' with pkg-config executable '/home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/bin/pkgconf'
env[PKG_CONFIG_LIBDIR]: /home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/aarch64-buildroot-linux-gnu/sysroot/usr/lib/pkgconfig:/home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/aarch64-buildroot-linux-gnu/sysroot/usr/share/pkgconfig
env[PKG_CONFIG_PATH]:
env[PKG_CONFIG_SYSROOT_DIR]: /home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/aarch64-buildroot-linux-gnu/sysroot
Called `/home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/bin/pkgconf --modversion python-3.10` -> 0
3.10
env[PKG_CONFIG_LIBDIR]: /home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/aarch64-buildroot-linux-gnu/sysroot/usr/lib/pkgconfig:/home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/aarch64-buildroot-linux-gnu/sysroot/usr/share/pkgconfig
env[PKG_CONFIG_PATH]:
env[PKG_CONFIG_SYSROOT_DIR]: /home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/aarch64-buildroot-linux-gnu/sysroot
Called `/home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/bin/pkgconf --cflags python-3.10` -> 0
-I/home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/aarch64-buildroot-linux-gnu/sysroot/usr/include/python3.10
env[PKG_CONFIG_LIBDIR]: /home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/aarch64-buildroot-linux-gnu/sysroot/usr/lib/pkgconfig:/home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/aarch64-buildroot-linux-gnu/sysroot/usr/share/pkgconfig
env[PKG_CONFIG_ALLOW_SYSTEM_LIBS]: 1
env[PKG_CONFIG_PATH]:                                                                                                                         
env[PKG_CONFIG_SYSROOT_DIR]: /home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/aarch64-buildroot-linux-gnu/sysroot
Called `/home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/bin/pkgconf --libs python-3.10` -> 0

env[PKG_CONFIG_LIBDIR]: /home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/aarch64-buildroot-linux-gnu/sysroot/usr/lib/pkgconfig:/home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/aarch64-buildroot-linux-gnu/sysroot/usr/share/pkgconfig
env[PKG_CONFIG_PATH]: 
env[PKG_CONFIG_SYSROOT_DIR]: /home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/aarch64-buildroot-linux-gnu/sysroot
Called `/home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/bin/pkgconf --libs python-3.10` -> 0

Running command: /home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/bin/python3 -c import os; os.chdir(".."); import numpy; print(numpy.get_include())
--- stdout ---
/home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/lib/python3.10/site-packages/numpy/core/include

libnpymath.a is present on the target:

$ find . -type f -name "libnpymath.a"./output/target/usr/lib/python3.10/site-packages/numpy/core/lib/libnpymath.a
./output/host/aarch64-buildroot-linux-gnu/sysroot/usr/lib/python3.10/site-packages/numpy/core/lib/libnpymath.a
./output/host/lib/python3.10/site-packages/numpy/core/lib/libnpymath.a
./output/build/host-python-numpy-1.21.2/build/temp.linux-x86_64-3.10/libnpymath.a
./output/build/python-numpy-1.21.2/build/temp.linux-aarch64-3.10/libnpymath.a

I think one confusing thing is that we use python for code generation tasks (in which case host python - in Buildroot’s terminology, so on the build machine - is fine), for obtaining paths to source code like the f2py and pybind11 sources (in which case host python and host f2py/pybind11 is also fine), and for obtaining paths to two static libraries: libnpymath.a and libnpyrandom.a (those need to come from the target).

And finally we also use the py3.extension_module etc.; here py3 is the host Python but Meson knows it’s cross-compiling and does its thing to build extensions for the target Python, as can be seen in the detailed log above from Meson looking for python-3.10 via pkg-config and then including this in the compile flags:

-I/home/rgommers/code/tmp/buildroot-scipy/oscimp_br2_external/buildroot/output/host/aarch64-buildroot-linux-gnu/sysroot/usr/include/python3.10

Okay, so far so good. We just need to override the paths to the two static libraries we need it looks like.