BUG: signal.decimate returns unexpected values with float32 arrays

Describe your issue.

This may be a duplicate of gh-15072, but 1) I don’t have a big array (50 values), and 2) this method did not return any NaN’s, just unexpected large (negative or positive) values.

I finally caught the issue when I realized I had switched from scipy==1.4.1 to scipy==1.7.3 (before, it was working)

I’m attaching two plots made with the same code example, but each with a different scipy version. In the 1.4.1 one, the decimate result is correct and independent of the type of array, and in the 1.7.3, it’s wrong for the float32 array:

Reproducing Code Example

import scipy.signal as ssi
import matplotlib.pyplot as plt
import numpy as np

np.random.seed(1)
n = 50
p = np.random.rand(n)
q = 20
p_d1 = ssi.decimate(np.asarray(p, dtype='float32'), q)
p_d2 = ssi.decimate(np.asarray(p, dtype='float64'), q)
print('max p_d1 (float32): ', max(p_d1))  # scipy==1.4.1: 0.526 / scipy==1.7.3: 5279  <- wrong
print('max p_d2 (float64): ', max(p_d2))  # scipy==1.4.1: 0.526 / scipy==1.7.3: 0.526

# Optionally plot
t = list(range(n))
t_d = [t[i * q] for i in range(len(p_d1))]
plt.plot(t, p, marker='o')
plt.plot(t_d, p_d1, marker='s')
plt.plot(t_d, p_d2, marker='x')
plt.legend(['float32', 'float32 > decimated', 'float64 > decimated'], loc='upper right')
plt.title('scipy==1.4.1')  # scipy==1.4.1 / scipy==1.7.3 (latest to date)
plt.show()

Error message

None!!

SciPy/NumPy/Python version information

No issue with scipy==1.4.1; Issue with scipy==1.7.3

[roryyorke]

I think this is related to #13529 - we need to switch to second-order sections, see stalled #14371.

Here’s a not-heavily-tested sos_decimate that tends to create very high order filters. It handles this test OK, though not brilliantly (I think at least partly due to “boundary-condition” issues combined with the high order filters; maybe having #11205 would help?).

I’m not sure about using padlen=0, but it’s needed for this short input and, again, the high-order filters.

def sos_dist_gain(sos):
    # distribute gain by scale factors of 2
    sos = sos.copy()

    # idea is to make gain of each section approximately equal
    # express gain as k * 2 ** m
    k, m = np.frexp(sos[0][0])
    d, q = divmod(m, sos.shape[0])
    sos[0][:3] *= 2.0 ** (-m + d + q)
    for i in range(1, sos.shape[0]):
        sos[i][:3] *= 2.0 ** d
    return sos


def sos_decimate(x, n, ftype='butter'):
    x = np.asarray(x)
    sos = signal.iirdesign(0.9/n, 1/n, 1.5, 20,
                           ftype=ftype, output='sos')

    # make DC gain 1
    sos[0][:3] /= signal.sosfreqz(sos, [0])[1][0].real

    # distribute gains to avoid underflow of coeffs in
    # single precision
    sos = sos_dist_gain(sos)

    if x.dtype in [np.float32, np.complex128]:
        sos = sos.astype(np.float32)

    return signal.sosfiltfilt(sos, x, padlen=0)[::n]

Here’s an n=50 result (reproducing the above), then an n=500 result, which shows the boundary-condition effects:

[varjak]

Thanks for looking into it. I don’t think that decimals stored in float32 should be downsampled to values x10000 larger. So I think the algorithm is at fault here, and although converting the inputs to float64 to reconvert them back at the end might work, it may not be a good solution. In scipy v. 1.4.1 it appears to work, so could we not use the same algorithm?