BUG: SciPy test failures due to changes in NumPy main or Python 3.11

Describe the issue:

Some SciPy tests involving masked arrays have been failing since ~7/15 (e.g. see CI in scipy/scipy#16610). It looks like some of the failures are resolved by gh-21977, but there are some that remain.

=================================== FAILURES ===================================
______________________ TestCorr.test_kendalltau_seasonal _______________________
../testenv/lib/python3.11/site-packages/scipy/stats/tests/test_mstats_basic.py:405: in test_kendalltau_seasonal
    assert_almost_equal(output['global p-value (indep)'], 0.008, 3)
        output     = {'chi2 total': 0.9138890055275067, 'chi2 trend': 0.03518368100765739, 'global p-value (dep)': 0.9592698679548645, 'global p-value (indep)': 0.9415878180349753, ...}
        self       = <scipy.stats.tests.test_mstats_basic.TestCorr object at 0x7f75a3bf2650>
        x          = masked_array(
  data=[[--, 4.0, 3.0, --],
        [--, 3.0, 2.0, 6.0],
        [4.0, 5.0, 5.0, 11.0],
        [2.0, 3.... False, False, False],
        [False, False, False, False],
        [False, False,  True, False]],
  fill_value=1e+20)
/home/runner/.local/lib/python3.11/site-packages/numpy/ma/testutils.py:189: in assert_almost_equal
    raise AssertionError(msg)
E   AssertionError: 
E   Items are not equal:
E    ACTUAL: 0.9415878180349753
E    DESIRED: 0.008
        actual     = 0.9415878180349753
        decimal    = 3
        desired    = 0.008
        err_msg    = ''
        msg        = '\nItems are not equal:\n ACTUAL: 0.9415878180349753\n DESIRED: 0.008'
        verbose    = True
_______________________ TestTtest_rel.test_fully_masked ________________________
../testenv/lib/python3.11/site-packages/scipy/stats/tests/test_mstats_basic.py:1217: in test_fully_masked
    assert_array_equal(p, (np.nan, np.nan))
        outcome    = masked_array(
  data=[[--, -0.4495544480666782],
        [--, 0.6433801600172545],
        [--, 0.08038096427837299]],
  mask=[[ True, False],
        [ True, False],
        [ True, False]],
  fill_value=1e+20)
        p          = 1.0
        pair       = ([nan, nan], [1.0, 2.0])
        self       = <scipy.stats.tests.test_mstats_basic.TestTtest_rel object at 0x7f75a2eedb50>
        t          = masked
/home/runner/.local/lib/python3.11/site-packages/numpy/ma/testutils.py:225: in assert_array_equal
    assert_array_compare(operator.__eq__, x, y,
        err_msg    = ''
        verbose    = True
        x          = 1.0
        y          = (nan, nan)
/home/runner/.local/lib/python3.11/site-packages/numpy/ma/testutils.py:213: in assert_array_compare
    return np.testing.assert_array_compare(comparison,
        comparison = <built-in function eq>
        err_msg    = ''
        fill_value = True
        header     = 'Arrays are not equal'
        m          = False
        verbose    = True
        x          = masked_array(data=1.,
             mask=False,
       fill_value=1e+20)
        y          = masked_array(data=[nan, nan],
             mask=False,
       fill_value=1e+20)
/opt/hostedtoolcache/Python/3.11.0-beta.4/x64/lib/python3.11/contextlib.py:81: in inner
    return func(*args, **kwds)
E   AssertionError: 
E   Arrays are not equal
E   
E   x and y nan location mismatch:
E    x: array(1.)
E    y: array([nan, nan])
        args       = (<built-in function eq>, array(1.), array([nan, nan]))
        func       = <function assert_array_compare at 0x7f75b19c16c0>
        kwds       = {'err_msg': '', 'header': 'Arrays are not equal', 'verbose': True}
        self       = <contextlib._GeneratorContextManager object at 0x7f75b19a91d0>
_______________________ TestTtest_ind.test_fully_masked ________________________
../testenv/lib/python3.11/site-packages/scipy/stats/tests/test_mstats_basic.py:1316: in test_fully_masked
    assert_array_equal(p, (np.nan, np.nan))
        outcome    = masked_array(
  data=[[--, -0.4495544480666782],
        [--, 0.6433801600172545],
        [--, 0.08038096427837299]],
  mask=[[ True, False],
        [ True, False],
        [ True, False]],
  fill_value=1e+20)
        p          = 1.0
        pair       = (masked_array(data=[--, --, --],
             mask=[ True,  True,  True],
       fill_value=1e+20,
            dtype=f...1600172545,
                   0.08038096427837299],
             mask=[False, False, False],
       fill_value=1e+20))
        self       = <scipy.stats.tests.test_mstats_basic.TestTtest_ind object at 0x7f75a3837150>
        t          = masked
/home/runner/.local/lib/python3.11/site-packages/numpy/ma/testutils.py:225: in assert_array_equal
    assert_array_compare(operator.__eq__, x, y,
        err_msg    = ''
        verbose    = True
        x          = 1.0
        y          = (nan, nan)
/home/runner/.local/lib/python3.11/site-packages/numpy/ma/testutils.py:213: in assert_array_compare
    return np.testing.assert_array_compare(comparison,
        comparison = <built-in function eq>
        err_msg    = ''
        fill_value = True
        header     = 'Arrays are not equal'
        m          = False
        verbose    = True
        x          = masked_array(data=1.,
             mask=False,
       fill_value=1e+20)
        y          = masked_array(data=[nan, nan],
             mask=False,
       fill_value=1e+20)
/opt/hostedtoolcache/Python/3.11.0-beta.4/x64/lib/python3.11/contextlib.py:81: in inner
    return func(*args, **kwds)
E   AssertionError: 
E   Arrays are not equal
E   
E   x and y nan location mismatch:
E    x: array(1.)
E    y: array([nan, nan])
        args       = (<built-in function eq>, array(1.), array([nan, nan]))
        func       = <function assert_array_compare at 0x7f75b19c16c0>
        kwds       = {'err_msg': '', 'header': 'Arrays are not equal', 'verbose': True}
        self       = <contextlib._GeneratorContextManager object at 0x7f75b19a91d0>
______________________ TestTtest_1samp.test_fully_masked _______________________
../testenv/lib/python3.11/site-packages/scipy/stats/tests/test_mstats_basic.py:1403: in test_fully_masked
    assert_array_equal(p, expected)
        expected   = (nan, nan)
        outcome    = masked_array(data=[--, --, --],
             mask=[ True,  True,  True],
       fill_value=1e+20,
            dtype=float64)
        p          = 1.0
        pair       = ((nan, nan), 0.0)
        self       = <scipy.stats.tests.test_mstats_basic.TestTtest_1samp object at 0x7f75a35bfa50>
        sup        = <numpy.testing._private.utils.suppress_warnings object at 0x7f75996c5150>
        t          = masked
/home/runner/.local/lib/python3.11/site-packages/numpy/ma/testutils.py:225: in assert_array_equal
    assert_array_compare(operator.__eq__, x, y,
        err_msg    = ''
        verbose    = True
        x          = 1.0
        y          = (nan, nan)
/home/runner/.local/lib/python3.11/site-packages/numpy/ma/testutils.py:213: in assert_array_compare
    return np.testing.assert_array_compare(comparison,
        comparison = <built-in function eq>
        err_msg    = ''
        fill_value = True
        header     = 'Arrays are not equal'
        m          = False
        verbose    = True
        x          = masked_array(data=1.,
             mask=False,
       fill_value=1e+20)
        y          = masked_array(data=[nan, nan],
             mask=False,
       fill_value=1e+20)
/opt/hostedtoolcache/Python/3.11.0-beta.4/x64/lib/python3.11/contextlib.py:81: in inner
    return func(*args, **kwds)
E   AssertionError: 
E   Arrays are not equal
E   
E   x and y nan location mismatch:
E    x: array(1.)
E    y: array([nan, nan])
        args       = (<built-in function eq>, array(1.), array([nan, nan]))
        func       = <function assert_array_compare at 0x7f75b19c16c0>
        kwds       = {'err_msg': '', 'header': 'Arrays are not equal', 'verbose': True}
        self       = <contextlib._GeneratorContextManager object at 0x7f75b19a91d0>
_________________________ test_chisquare_masked_arrays _________________________
../testenv/lib/python3.11/site-packages/scipy/stats/tests/test_stats.py:3579: in test_chisquare_masked_arrays
    mat.assert_array_almost_equal(g, expected_g, decimal=15)
        chi2       = <scipy.stats._continuous_distns.chi2_gen object at 0x7f75ad58ae50>
        chisq      = masked_array(data=[24.0, 0.5],
             mask=[False, False],
       fill_value=1e+20)
        expected_chisq = array([24. ,  0.5])
        expected_g = array([22.18070978,  0.50534308])
        g          = array([nan, nan])
        mask       = array([[0, 1],
       [0, 1],
       [0, 0],
       [0, 0],
       [1, 0]])
        mobs       = masked_array(
  data=[[8, --],
        [8, --],
        [16, 3],
        [32, 4],
        [--, 5]],
  mask=[[False,  T...,
        [False,  True],
        [False, False],
        [False, False],
        [ True, False]],
  fill_value=999999)
        obs        = array([[ 8, -1],
       [ 8, -1],
       [16,  3],
       [32,  4],
       [-1,  5]])
        p          = array([nan, nan])
/home/runner/.local/lib/python3.11/site-packages/numpy/ma/testutils.py:265: in assert_array_almost_equal
    assert_array_compare(compare, x, y, err_msg=err_msg, verbose=verbose,
        compare    = <function assert_array_almost_equal.<locals>.compare at 0x7f758bbc84a0>
        decimal    = 15
        err_msg    = ''
        verbose    = True
        x          = array([nan, nan])
        y          = array([22.18070978,  0.50534308])
/home/runner/.local/lib/python3.11/site-packages/numpy/ma/testutils.py:213: in assert_array_compare
    return np.testing.assert_array_compare(comparison,
        comparison = <function assert_array_almost_equal.<locals>.compare at 0x7f758bbc84a0>
        err_msg    = ''
        fill_value = True
        header     = 'Arrays are not almost equal'
        m          = False
        verbose    = True
        x          = masked_array(data=[nan, nan],
             mask=False,
       fill_value=1e+20)
        y          = masked_array(data=[22.18070978,  0.50534308],
             mask=False,
       fill_value=1e+20)
/opt/hostedtoolcache/Python/3.11.0-beta.4/x64/lib/python3.11/contextlib.py:81: in inner
    return func(*args, **kwds)
E   AssertionError: 
E   Arrays are not almost equal
E   
E   x and y nan location mismatch:
E    x: array([nan, nan])
E    y: array([22.18071 ,  0.505343])
        args       = (<function assert_array_almost_equal.<locals>.compare at 0x7f758bbc84a0>, array([nan, nan]), array([22.18070978,  0.50534308]))
        func       = <function assert_array_compare at 0x7f75b19c16c0>
        kwds       = {'err_msg': '', 'header': 'Arrays are not almost equal', 'verbose': True}
        self       = <contextlib._GeneratorContextManager object at 0x7f75b19a91d0>
============================= slowest 10 durations =============================
16.80s call     build/testenv/lib/python3.11/site-packages/scipy/stats/tests/test_continuous_basic.py::test_kappa4_array_gh13582
12.20s call     build/testenv/lib/python3.11/site-packages/scipy/stats/tests/test_continuous_basic.py::test_cont_basic[500-200-skewnorm-arg91]
9.72s call     build/testenv/lib/python3.11/site-packages/scipy/optimize/tests/test_direct.py::TestDIRECT::test_segmentation_fault[False]
7.68s call     build/testenv/lib/python3.11/site-packages/scipy/sparse/linalg/_isolve/tests/test_iterative.py::test_precond_inverse[case1]
7.33s call     build/testenv/lib/python3.11/site-packages/scipy/_lib/tests/test_import_cycles.py::test_modules_importable
5.79s call     build/testenv/lib/python3.11/site-packages/scipy/optimize/tests/test_lsq_linear.py::TestTRF::test_large_rank_deficient
5.76s call     build/testenv/lib/python3.11/site-packages/scipy/optimize/tests/test_lsq_linear.py::TestBVLS::test_large_rank_deficient
4.71s call     build/testenv/lib/python3.11/site-packages/scipy/stats/tests/test_continuous_basic.py::test_cont_basic[500-200-truncweibull_min-arg100]
3.93s call     build/testenv/lib/python3.11/site-packages/scipy/optimize/tests/test_optimize.py::TestOptimizeSimple::test_minimize_callback_copies_array[fmin]
3.68s call     build/testenv/lib/python3.11/site-packages/scipy/optimize/_trustregion_constr/tests/test_report.py::test_gh12922
=========================== short test summary info ============================
FAILED ../testenv/lib/python3.11/site-packages/scipy/stats/tests/test_mstats_basic.py::TestCorr::test_kendalltau_seasonal
FAILED ../testenv/lib/python3.11/site-packages/scipy/stats/tests/test_mstats_basic.py::TestTtest_rel::test_fully_masked
FAILED ../testenv/lib/python3.11/site-packages/scipy/stats/tests/test_mstats_basic.py::TestTtest_ind::test_fully_masked
FAILED ../testenv/lib/python3.11/site-packages/scipy/stats/tests/test_mstats_basic.py::TestTtest_1samp::test_fully_masked
FAILED ../testenv/lib/python3.11/site-packages/scipy/stats/tests/test_stats.py::test_chisquare_masked_arrays
= 5 failed, 36748 passed, 2162 skipped, [1211](https://github.com/scipy/scipy/runs/7379137176?check_suite_focus=true#step:7:1212)4 deselected, 139 xfailed, 7 xpassed in 538.49s (0:08:58) =

I am having trouble creating a MWE involving only NumPy because I haven’t been successful at building NumPy main. I hope someone with (any relatively modern version of) SciPy and NumPy main can help me find one.

This test below passes in NumPy 1.21.2, but I believe it will fail with NumPy main. Can someone confirm and tell me what the value of res is in NumPy main?

Reproduce the code example:

import numpy as np
from numpy import ma
from scipy.stats import mstats
outcome = ma.masked_array(np.random.randn(3, 2), mask=[[1, 0], [1, 0], [1, 0]])
res = mstats.ttest_rel(outcome[:, 0], outcome[:, 1])
# I don't think it is good practice to compare a masked element to a NaN,
# but this is what was in the SciPy test.
np.testing.assert_array_equal(res, (np.nan, np.nan))
# In any case, if this no longer passes, why not? 
# What is the value of `res` using NumPy main?
print(res)

Error message:

-

NumPy/Python version information:

1.21.2 3.9.7 (default, Sep 16 2021, 16:59:28) [MSC v.1916 64 bit (AMD64)]

[mdhaber]

OK here we go. It doesn’t just involve NumPy, unfortunately, but there is a behavior that was changed recently:

import numpy as np
from scipy import special
df = np.ma.array([1])
t = np.ma.masked
special.stdtr(df, t)

This used to produce a masked array with the only element masked; now it produces an unmasked array([0.5]).

A little simpler:

import numpy as np
import scipy.special
special.ndtr(np.ma.masked)  # was `masked`, now `0.5`

[mdhaber]

Thanks. As I mentioned in the OP, I wasn’t sure whether the snippet would reproduce the issue or not. Now that I know that it does not, perhaps I can dig a little more to see what is really going on.