Adding forecasting backtest/evaluate function

Is your feature request related to a problem? Please describe. I would like to add a backtesting class. We should ideally then also implement an expanding window splitter functionality as discussed in #552 . But the Backtest would just receive a splitter as an argument.

Describe the solution you’d like A class like Backtest (naming to be discussed) with a signature like this:

class Backtest(object):
    """Backtest forecasters

    Parameters
    ----------
    forecaster: estimator object
    splitter: splitter object
        e.g. SlidingWindowSplitter
    score: function
        A function like smape_loss that has y_pred, y_test as arguments and returns a float

    Returns
    ----------
    error: float
        Average error of fitted estimators calculated by means of the score function
    """

The class should have a fit() function like a regular estimator object. I guess a predict() function does not make sense here. The class will be able to also take a forecaster of type ForecastingGridSearchCV, as in production people might refit and tune the model by progress in time.

Optionally, the class can print the number of fits, like in ForecastingGridSearchCV when fitting.

I am not sure on how to structure the return and what else should be returned, I guess just return self and have the rest as class variables? We could do sth like in ForecastingGridSearchCV with the cv_results_ variable, so we could also store there the errors, predictions and also the params of each iteration (because params might differ when using a grid search as forecaster).

Where should a class Backtest be placed in the sktime class structure? I propose sktime/sktime/forecasting/model_backtest/_backtest.py.

[aiwalter]

Ok, lets go then for now for the function as I am also a fan of keeping things simple. The plot is actually a nice to have, and might be just added to a notebook and not be part of the sktime packe. Anyway, I might try to add an optional return argument to return the data that is required for the plot.

[aiwalter]

@fkiraly at Mercedes we say “The best or nothing”, and its all about luxury 😄 . And sktime fits to this slogan as well. I would appreciate going for a class. We are more flexible in what we return, as we can store stuff just in self and the user can pick what he wants to.