NumPy ABI does not successfully maintain forward compatibility -- should it?

Not sure whether this is something we consider a bug, but flagging for potential discussion and so it doesn’t get lost: I hadn’t realized until today that NumPy in practice does not provide a forward compatible ABI, i.e., if you build with numpy 1.9 then try to run against 1.8, this may not work. Apparently packages that care about this like sklearn are actively working around this by carefully installing old versions of numpy before building wheels.

In particular, we have several times added extra fields to the dtype struct. In practice this is normally fine b/c no-one actually accesses these fields, but Cython in particular does struct size checking. For backwards compat – build against 1.8 and then run against 1.9 – the struct appears to get larger, and Cython merely issues a warning (which we suppress). For forward compat – build against 1.9 and then run against 1.8 – the struct appears to get smaller, and in this case Cython issues a hard error.

We could work around this by simply exposing a truncated struct to user code, so that Cython sees a small struct when doing sizeof, and the actual object is always larger then this, meaning that we always hit the warning path rather than the error path.

I don’t know if this is the only problem we would have to fix in order to achieve forward compatibility, e.g. I haven’t checked the C API import code to see what import_multiarray or import_umath do when they find themselves running against an older version of numpy.

If we want to take ABI compatibility seriously I guess we should probably also start requiring C API users to explicitly state which version of the ABI they expect, and enforce that they don’t get access to anything newer than that. This would at least give us the option then in the future to provide different versions of the same function to old-users and new-users.

[seberg]

Closing. This is effectively solved by https://pypi.org/project/oldest-supported-numpy/ and build requires. So we don’t attempt this, and thats that. (There is an interesting problem of ensuring that API break in the future is adhered to even though you compile against an older version, i.e. some compat headers, etc. This would be for a major NumPy 2.0 release, and something we need to figure out… But probably not on this issue.)

[mattip]

If I recall correctly, SciPy pins its build system to the oldest numpy it wishes to support, but tests against later numpy versions.