On PyPy + windows, complex quadpack seem inconsistent

xref conda-forge/scipy-feedstock#196

There are 144 test failures when I try to build scipy 1.7.2 and 1.7.0 with PyPy3.7 on windows. The all are in the TestStudentizedRange.test_pdf_against_mp tests and look like the the failure below. The test warns IntegrationWarning: The occurrence of roundoff error is detected, which prevents the requested tolerance from being achieved. The error may be underestimated. Could someone give me a hint where to start looking for the problem? I am building for now with set SCIPY_USE_PYTHRAN=0 but I don’t think that is connected at all. The problem is most likely with PyPy, but I don’t understand the code enough to find a hook to start debugging this. What is special about this test set?

TestStudentizedRange.test_pdf_against_mp

___________________ TestStudentizedRange.test_pdf_against_mp[case_result54] ________
scipy\stats\tests\test_distributions.py:4572: in test_pdf_against_mp
    res = stats.studentized_range.pdf(*qkv)
        case_result = {'mp_result': 0.43334135870667473, 'src_case': {'expected_atol': 1e-11, 'expected_rtol': 1e-11, 'k': 20, 'q': 4, ...}}
        mp_result  = 0.43334135870667473
        qkv        = (4, 20, 50)
        self       = <scipy.stats.tests.test_distributions.TestStudentizedRange object at 0x0000027796144368>
        src_case   = {'expected_atol': 1e-11, 'expected_rtol': 1e-11, 'k': 20, 'q': 4, ...}
scipy\stats\_distn_infrastructure.py:1879: in pdf
    place(output, cond, self._pdf(*goodargs) / scale)
        args       = (array(20), array(50))
        cond       = True
        cond0      = True
        cond1      = True
        dtyp       = dtype('float64')
        goodargs   = [array([4.]), array([20]), array([50])]
        kwds       = {}
        loc        = array(0)
        output     = array(0.)
        scale      = array([1])
        self       = <scipy.stats._continuous_distns.studentized_range_gen object at 0x000002778e095cc8>
        x          = array(4.)
scipy\stats\_continuous_distns.py:9492: in _pdf
    return np.float64(ufunc(x, k, df))
        _single_pdf = <function studentized_range_gen._pdf.<locals>._single_pdf at 0x00000277959a2020>
        cython_symbol = '_studentized_range_pdf'
        df         = array([50])
        k          = array([20])
        self       = <scipy.stats._continuous_distns.studentized_range_gen object at 0x000002778e095cc8>
        ufunc      = <ufunc '? (vectorized)'>
        x          = array([4.])
scipy\stats\_continuous_distns.py:9489: in _single_pdf
    return integrate.nquad(llc, ranges=ranges, opts=opts)[0]
        arg        = [4.0, 20, 50, 32.32048465687308]
        cython_symbol = '_studentized_range_pdf'
        df         = 50
        k          = 20
        llc        = LowLevelCallable(<capsule object "double (int, double *, void *)" at 0x0000027783549D80>, c_void_p(2712581508336))
        log_const  = 32.32048465687308
        opts       = {'epsabs': 1e-11, 'epsrel': 1e-12}
        q          = 4.0
        ranges     = [(-inf, inf), (0, inf)]
        usr_data   = c_void_p(2712581508336)
scipy\integrate\quadpack.py:825: in nquad
    return _NQuad(func, ranges, opts, full_output).integrate(*args)
        args       = ()
        depth      = 2
        full_output = False
        func       = LowLevelCallable(<capsule object "double (int, double *, void *)" at 0x0000027783549D80>, c_void_p(2712581508336))
        opts       = [<scipy.integrate.quadpack._OptFunc object at 0x00000277961441a8>, <scipy.integrate.quadpack._OptFunc object at 0x00000277961441a8>]
        ranges     = [<scipy.integrate.quadpack._RangeFunc object at 0x0000027796144170>, <scipy.integrate.quadpack._RangeFunc object at 0x0000027796144288>]
scipy\integrate\quadpack.py:880: in integrate
    **opt)
        args       = ()
        depth      = 0
        f          = functools.partial(<bound method _NQuad.integrate of <scipy.integrate.quadpack._NQuad object at 0x0000027796144138>>, depth=1)
        fn_opt     = <scipy.integrate.quadpack._OptFunc object at 0x00000277961441a8>
        fn_range   = <scipy.integrate.quadpack._RangeFunc object at 0x0000027796144288>
        high       = inf
        ind        = -1
        kwargs     = {}
        low        = 0
        opt        = {'epsabs': 1e-11, 'epsrel': 1e-12}
        self       = <scipy.integrate.quadpack._NQuad object at 0x0000027796144138>
scipy\integrate\quadpack.py:352: in quad
    points)
        a          = 0
        args       = ()
        b          = inf
        epsabs     = 1e-11
        epsrel     = 1e-12
        flip       = False
        full_output = False
        func       = functools.partial(<bound method _NQuad.integrate of <scipy.integrate.quadpack._NQuad object at 0x0000027796144138>>, depth=1)
        limit      = 50
        limlst     = 50
        maxp1      = 50
        points     = None
        weight     = None
        wopts      = None
        wvar       = None
scipy\integrate\quadpack.py:465: in _quad
    return _quadpack._qagie(func,bound,infbounds,args,full_output,epsabs,epsrel,limit)
        a          = 0
        args       = ()
        b          = inf
        bound      = 0
        epsabs     = 1e-11
        epsrel     = 1e-12
        full_output = False
        func       = functools.partial(<bound method _NQuad.integrate of <scipy.integrate.quadpack._NQuad object at 0x0000027796144138>>, depth=1)
        infbounds  = 1
        limit      = 50
        points     = None
..\..\..\..\pypy3.7-v7.3.7-win64\lib_pypy\_functools.py:80: in __call__
    return self._func(*(self._args + fargs), **fkeywords)
        fargs      = (-1.0,)
        fkeywords  = {'depth': 1}
        self       = functools.partial(<bound method _NQuad.integrate of <scipy.integrate.quadpack._NQuad object at 0x0000027796144138>>, depth=1)
scipy\integrate\quadpack.py:880: in integrate
    **opt)
        args       = (-1.0,)
        depth      = 1
        f          = LowLevelCallable(<capsule object "double (int, double *, void *)" at 0x0000027783549D80>, c_void_p(2712581508336))
        fn_opt     = <scipy.integrate.quadpack._OptFunc object at 0x00000277961441a8>
        fn_range   = <scipy.integrate.quadpack._RangeFunc object at 0x0000027796144170>
        high       = inf
        ind        = -2
        kwargs     = {}
        low        = -inf
        opt        = {'epsabs': 1e-11, 'epsrel': 1e-12}
        self       = <scipy.integrate.quadpack._NQuad object at 0x0000027796144138>
scipy\integrate\quadpack.py:400: in quad
    warnings.warn(msg, IntegrationWarning, stacklevel=2)
E   scipy.integrate.quadpack.IntegrationWarning: The occurrence of roundoff error is detected, which prevents
E     the requested tolerance from being achieved.  The error may be
E     underestimated.
        a          = -inf
        args       = (-1.0,)
        b          = inf
        epsabs     = 1e-11
        epsrel     = 1e-12
        flip       = False
        full_output = False
        func       = LowLevelCallable(<capsule object "double (int, double *, void *)" at 0x0000027783549D80>, c_void_p(2712581508336))
        ier        = 2
        limit      = 50
        limlst     = 50
        maxp1      = 50
        msg        = 'The occurrence of roundoff error is detected, which prevents \n  the requested tolerance from being achieved.  The error may be \n  underestimated.'
        msgs       = {80: 'A Python error occurred possibly while calling the function.', 1: 'The maximum number of subdivisions (50) has b...\n  underestimated.', 3: 'Extremely bad integrand behavior occurs at some points of the\n  integration interval.', ...}
        points     = None
        retval     = (nan, nan, 2)
        weight     = None
        wopts      = None
        wvar       = None

</detail>

[mattip]

I have a smaller reproducer in https://gist.github.com/mattip/346234bc51a8a3afca10ea8929a53df0. The problem only happens when PyPy JIT kicks in: eventually the python callback in dqegie is called with -1 as an argument instead of a valid value. I am trying to work out how that happens. I have ruled out FPU precision by checking _control87. It may be some kind of argument passing bug.

One thing is strange. When building scipy, I see warnings that small may be used uninitialized https://github.com/scipy/scipy/blob/62f75fa33f1a832043b53f0694693dee0fe9503a/scipy/integrate/quadpack/dqagie.f#L411

That code seems to be 21 years old, so it must have been vetted by now. Is there a problem or is it just my fortran skills are rusty?

scipy\integrate\quadpack\dqagie.f:411:0:

         small = small*0.5d+00
 
Warning: 'small' may be used uninitialized in this function [-Wmaybe-uninitialized]
scipy\integrate\quadpack\dqagie.f:372:0:

    40   if(ierro.eq.3.or.erlarg.le.ertest) go to 60
 
Warning: 'ertest' may be used uninitialized in this function [-Wmaybe-uninitialized]
scipy\integrate\quadpack\dqagie.f:362:0:

         erlarg = erlarg-erlast
 
Warning: 'erlarg' may be used uninitialized in this function [-Wmaybe-uninitialized]
scipy\integrate\quadpack\dqagie.f:425:0:

       if(ierro.eq.3) abserr = abserr+correc
 

[DominicChm]

Good idea! Along those lines, I figured out how to wrap the Cython to inspect inputs/outputs and extracted the distribution’s implementation for testing. Good news - the problem is still there.

Here’s the output of my test batch script.

====== COMPARE BOTH =======
This will take a while because results are written to file.
========= PYTHON3 =========
Moment (SCIPY):  1.8342745127993187
Moment (CUSTOM): 1.8342745127993187
PDF (SCIPY):  0.39877202027575265
PDF (CUSTOM): 0.39877202027575265
PDF eval 1: 9.600101461805174e-26
PDF eval 2: 0.018692499086136082

========= PYPY3 =========
Moment (SCIPY):  1.8482537701109825
Moment (CUSTOM): 1.8455557977539785
PDF (SCIPY):  0.39877202027575265
PDF (CUSTOM): 0.39368392908702027
PDF eval 1: 9.600101461805174e-26
PDF eval 2: 0.018692499086136082

========= COMPARE =========
Comparing element-wise...
Done!
Press any key to continue . . .

The CUSTOM output uses the extracted implementation, and is compiled by each respective interpreter. It should match SCIPY because they’re effectively same.

When compiled/ran by pypy, both the PDF and moment come out wrong vs the reference CPython run. What’s very interesting is that - despite the garbage output - the PyPy and CPython PDF integrands (the cython code) evaluate to the exact same values over a very large input set (the "Compare) section). That seems to rule out a studentized range compilation problem to me, at least for my problem. If you want to check what happens in your environment, I dumped the code here https://github.com/DominicChm/IntegrandTest. The above output is from compare.bat