question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

`backtest()`: Retrain every n steps

See original GitHub issue

Recently found your library and it is really great and handy! The evaluation of the forecasting performance is best done using a cross-validation approach. The backtest_forecasting()-function does that - although it currently iterates and re-trains the model on every single time step. In my application, I am training ten-thousands of different time series and it becomes computationally unfeasible to retrain on every time step. Another approach already implemented in scikit-learn is TimeSeriesSplit(), which generates a k-fold cross-validation split designed for time series. https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html

One solution to the issue would be to add support for different cross-validation techniques implemented in scikit-learn (such as TimeSeriesSplit()). Another solution I see could be to add a stride parameter in backtest_forecasting() that specifies how many time steps it should skip while backtesting.

Issue Analytics

  • State:open
  • Created 3 years ago
  • Reactions:1
  • Comments:10 (5 by maintainers)

github_iconTop GitHub Comments

4reactions
hrzncommented, Jul 11, 2020

Thanks for pointing this out @aschl. We are currently re-working quite substantially the backtesting functionality, and indeed our plan is to introduce a stride parameter like you suggest. We will also introduce moving windows for the training set (instead of expending only, as now). These should come in a release sometime soon.

1reaction
hrzncommented, Nov 2, 2022

You’re right @deltamacht. Gridsearch is still pretty basic… There’s however a stride parameter in gridsearch() which maybe goes some way into doing what you’re looking for.

Read more comments on GitHub >

github_iconTop Results From Across the Web

How To Backtest Machine Learning Models for Time Series ...
Walk forward validation is a method for estimating the skill of the model on out of sample data. We contrive out of sample...
Read more >
Baseline Models — darts documentation
This process is accomplished by using the backtest() function as a subroutine to produce historic forecasts starting from start that are compared against...
Read more >
Time series implementation details (Watson Machine Learning)
Each pipeline is trained on every allocated subset of training data and evaluated with ... In this step, the winning pipelines N are...
Read more >
How to parallelize a backtesting for a trading strategy based ...
At first, this is not parallelizing the process scheduling. If in doubts, one may revisit theory details on SERIAL, PARALLEL and ...
Read more >
Backtest — orbit 1.1.4dev documentation
The way to gauge the performance of a time-series model is through re-training models with different historic periods and check their forecast within...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found