autocorr vs autocorrplot

This issue is actually more of a question, it is quite possible that we can resolve this without actual code changes. arviz maintains autocorr code in stats.diagnostics._autocorr and plots.autocorrplot (both are inherited from the pymc3 codebase). Both here and in pymc3 the way the autocorrelation is computed differs, in particular plots.autocorrplot uses np.correlate(y, y, mode=2) whereas stats.diagnostics.autocorr uses fftconvolve(y, y[::-1]). Is there a reason why plots.autocorrplot does not simply use stats.diagnostics.autocorr under the hood?

[ColCarroll]

Ooh, maybe the opposite of historical reasons. I believe I was porting this to use xarray, and wanted to avoid plt.acorr, which used to be used: it does not add much to the code complexity, but allows a lot more flexibility in signature and styling. I did not even notice az.diagnostics._autocorr!

Checking out benchmarks and actual values, it looks like the two functions are not identical, but are quite close where there is actual autocorrelation, and that az.diagnostics._autocorr is much faster for large inputs, and modestly slower for small ones. I attached the code below. Trying np.correlate on 1e6 points scares my computer.

def numpy_version(y):
    points = len(y)
    z = np.correlate(y, y, mode=2)[points - 1:]
    return z / z[0]

metrics = []
for j in range(2, 6):
    points = 10 ** j
    y = np.random.randn(points)
    for lag in (0, 1, 10, 20):
        metrics.append({'points': points, 'lag': lag})
        z = y.copy()
        if lag:
            z[:-lag] = z[:-lag] + z[lag:]
        t0 = time.time()
        a = numpy_version(z)
        metrics[-1]['numpy_version'] = time.time() - t0
        t0 = time.time()
        b = az.diagnostics._autocorr(z)
        metrics[-1]['arviz_version'] = time.time() - t0
        metrics[-1]['difference'] = np.linalg.norm(a - b)

[MFreidank]

@junpenglao yes it is from pymc3. I noticed that it gives the same results as pystan. Thanks for providing background on the implementation and theory. I think it makes sense to use this method to compute autocorrelation in az.plots.autocorrplot as well. I will provide a pull request for that tomorrow.