speedup proposal: no lead loop in HindcastEnsemble but lead and init in obs

• create init and lead dimensions for observation seamlessly with cfsv2_ds["o"] = (( "L", "S"), o.sel(T=T))
• this approach is thinking more in real_time or valid_time, however, we do not have a dimension for time, but just a multi-dimensional coordinate time. why is this important? if were dimension, shape would get really large and get us faster into memory issues

Demo: https://gist.github.com/aaronspring/9d724cf385c1b29a5288eabf3e55148b

Idea comes from https://github.com/mktippett/ENSO/blob/master/ForecastVerification.ipynb Comparison of this approach with current climpred: 10x less tasks: https://gist.github.com/aaronspring/fa99abbee189d65305179d3344e0a405

Small summary:

• new verify reproduces verify from climpredv2.1.1 (tiny errors O(10-7))
• tasks decrease from ~100 to 13-31 (when applying to small data and do.chunk())
• timing reduces especially for same_verif
• timing increases for big geospatial data: 1000x1000 gridcells
• allows lead units from microseconds to years (also now independent of YS also Y or MS/M, QS/S)

Implementation proposal:

• add multi-dim coord time(init, lead) when instantiating initialized: faciliates PredictionEnsemble.plot() but needs to allow time as coord which is not explicitly not allowed
• second step: change alignment. Maybe not all the way as proposed here but maybe a combination of the new and old way, e.g. using the new lead time matrix but still looping over each lead?

Benchmark done on my 2018 macbookpro. results in csvfile in gist

Figure summary:

• Visualization of alignment: here remove observations in the 1990s partly
• timing reduces especially for same_verif
• timing increases for big geospatial data: 200x200 gridcells

[ahuang11]

Nevermind; “zero is a zero order spline. It’s value at any point is the last raw value seen.” not what I thought https://stackoverflow.com/questions/27698604/what-do-the-different-values-of-the-kind-argument-mean-in-scipy-interpolate-inte

If I am not mistaken, interp(method=‘zero’) is most robust for except clause if you don’t want it to be interpolated:

import numpy as np
import pandas as pd
import xarray as xr

cfs_url = 'http://iridl.ldeo.columbia.edu/SOURCES/.Models/.NMME/.NCEP-CFSv2/.HINDCAST/.MONTHLY/.T/dods'
obs_url = "http://iridl.ldeo.columbia.edu/expert/SOURCES/.NOAA/.NCEP/.EMC/.CMB/.GLOBAL/.Reyn_SmithOIv2/.monthly/.sst/T/1+index/dods"

obs = xr.open_dataset(obs_url, decode_times=False).isel(T=slice(0, 10))
cfs = xr.open_dataset(cfs_url, decode_times=False)

obs.interp(T=cfs['T'], method='zero')

[aaronspring]

I see this my way: once people are getting memory issues, they should/need to learn how to use dask to overcome this challenge. if they are way below memory, I dont mind how they do it.