`backtest()`: Retrain every n steps
See original GitHub issueRecently found your library and it is really great and handy!
The evaluation of the forecasting performance is best done using a cross-validation approach. The backtest_forecasting()
-function does that - although it currently iterates and re-trains the model on every single time step. In my application, I am training ten-thousands of different time series and it becomes computationally unfeasible to retrain on every time step. Another approach already implemented in scikit-learn is TimeSeriesSplit()
, which generates a k-fold cross-validation split designed for time series.
https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html
One solution to the issue would be to add support for different cross-validation techniques implemented in scikit-learn (such as TimeSeriesSplit()). Another solution I see could be to add a stride
parameter in backtest_forecasting()
that specifies how many time steps it should skip while backtesting.
Issue Analytics
- State:
- Created 3 years ago
- Reactions:1
- Comments:10 (5 by maintainers)
Thanks for pointing this out @aschl. We are currently re-working quite substantially the backtesting functionality, and indeed our plan is to introduce a
stride
parameter like you suggest. We will also introduce moving windows for the training set (instead of expending only, as now). These should come in a release sometime soon.You’re right @deltamacht. Gridsearch is still pretty basic… There’s however a
stride
parameter ingridsearch()
which maybe goes some way into doing what you’re looking for.