test_bisplev_integer_overflow: Segmentation fault (core dumped)

There is a bug in _fitpackmodule.c:195, that results seg fault in later part of code (see gh-12401 for the initial description of this segfault).

Problem can be reproduced if that code is compiled with -O2/-O3 gcc flag. I have tested it on gcc 7.5 and 9.2 and the behavior is the same. If such optimization level is enabled, then after line 194 mxy = mx*my; have been executed, compilator wont count mxy/my from if statement in line 195, but instead it will place mx variable here. In such a case the integer overflow is not recognized but it exists. I checked an assembly code and none division of mxy/my has been performed.

Reproducing code example:

#include <stdio.h>
#include <stdlib.h>

int main(int argc, char* argv[])
{
    if(argc != 2)
        return 1;

    int mx, my, mxy;
    mx = my = atoi(argv[1]);
    mxy = mx*my;

    if(my != 0 && mxy/my != mx)
        printf("overflow\n");
    else
        printf("ok\n");

    return 0;
}

This is short example that shows the problem. Compile it with:

$ gcc -O2 test.c -o test

and run with value that do overflow (in case scipy the value is 2621440):

$ ./test 2621440

output: ok (but overflow occured) This will work only with no optimizations or with optimizations -O1. Solution: In case of this example it should be unsigned mx, my, mxy;.

Possible fix

This issue can be fixed with following patch:

diff -ruN scipy_orig/scipy/interpolate/src/_fitpackmodule.c scipy/scipy/interpolate/src/_fitpackmodule.c
--- scipy_orig/scipy/interpolate/src/_fitpackmodule.c	2020-07-02 18:20:35.014620855 +0200
+++ scipy/scipy/interpolate/src/_fitpackmodule.c	2020-07-02 18:10:09.638601016 +0200
@@ -191,7 +191,7 @@
     ny = ap_ty->dimensions[0];
     mx = ap_x->dimensions[0];
     my = ap_y->dimensions[0];
-    mxy = mx*my;
+    mxy = (unsigned)mx * (unsigned)my;
     if (my != 0 && mxy/my != mx) {
         /* Integer overflow */
         PyErr_Format(PyExc_RuntimeError,

[pv]

Or, maybe better to just cast to unsigned (maybe size_t), since that’s defined and the operands should anyway be positive here.

[pv]

I think adding volatile is not the right fix — signed integer overflow is undefined behavior, so the compiler is still allowed to assume anything about the result.

The overflow should be checked before doing it, https://stackoverflow.com/questions/199333/how-do-i-detect-unsigned-integer-multiply-overflow For example, https://github.com/numpy/numpy/blob/a87ee593c3f5da5dcb42a8033ecbab0f0c9b7903/numpy/core/src/common/npy_extint128.h#L40-L56