[ENH] Past predictions over train set

Is your feature request related to a problem? Please describe.

I was wondering why can’t we get past predictions (by that I mean predictions made by the model on the train set) in sktime, actually one of the most fundamental ways to compare models is to compare their performance on train set relatively to test set performance and see if they’re over-fitting: if the model performance on train set is way better than it is one the test/validation set. But in sktime we can only (I guess) see future predictions because of the fh argument that starts at 1 (as the first prediction after train set) which means we can only compare y_pred to y_test.

Describe the solution you’d like

I think this needs a refactoring of how forecasters treat the ForecastHorizon. either forecasters should be able to accept negative values in their fh argument (kinda off) or maybe (as in statsmodels) the new forecasts should start at len(y_train).

Additional context

This makes a big difference for models that can be overfitted aka models with lots of hyper-parameters such as prophet where adding more and more seasonalities with high order can make your future predictions worse but your past predictions better. Actually all models that allow multiple seasonalities (BATS and TBAT) can over-fit easily but with this you’ll be able to see it with your eyes.

[aiwalter]

Yes then just do insample prediction

[fkiraly]

@ilyasmoutawwakil, not sure whether in-sample forecasts are what you are looking fore - those shouldn’t be used for backtesting!