AutoGluon stacking quality

What is the reason why AutoGluon shows significantly worse result compare to simple one layer stacking? 5 hours best_quality training gives private score 0.38131 compare to 0.37353 for 2 hours logistic regression.

[Innixma]

Re 1: This already exists ('LR' is the key to use in hyperparameters), however it is not used by default. I plan to add it as a default model in a future release (possibly in v0.3).

Re 2: We have a more sophisticated method of handling bagging / CV than is available in sklearn. We have to handle more complex cases than is supported by CalibratedClassifierCV. In future we may consider contributing the functionality back to sklearn, as this is some of the most important components of AutoGluon.

Re 3: It is impossible to pick the best model ahead of time for the test score. best is picked off of the strongest val score. ML would be very easy if we could know which model was best on the test data ahead of time, but that is not the case.

Re 4: We are happy to accept contributions! If you’d like, please open a PR which adds this model and we can test it / benchmark it to see if it improves upon our existing methods!

[Innixma]

You can set to 1000 trees via the hyperparameters argument in .fit: https://auto.gluon.ai/stable/_modules/autogluon/tabular/predictor/predictor.html#TabularPredictor.fit

something like:

hyperparameters = {
    'NN': {},
    'GBM': [
        {},
        {'extra_trees': True, 'ag_args': {'name_suffix': 'XT'}},
        'GBMLarge',
    ],
    'CAT': {},
    'XGB': {},
    'FASTAI': {},
    'RF': [
        {'criterion': 'gini', 'n_estimators': 1000, 'ag_args': {'name_suffix': 'Gini', 'problem_types': ['binary', 'multiclass']}},
        {'criterion': 'entropy', 'n_estimators': 1000, 'ag_args': {'name_suffix': 'Entr', 'problem_types': ['binary', 'multiclass']}},
        {'criterion': 'mse', 'n_estimators': 1000, 'ag_args': {'name_suffix': 'MSE', 'problem_types': ['regression']}},
    ],
    'XT': [
        {'criterion': 'gini', 'n_estimators': 1000, 'ag_args': {'name_suffix': 'Gini', 'problem_types': ['binary', 'multiclass']}},
        {'criterion': 'entropy', 'n_estimators': 1000, 'ag_args': {'name_suffix': 'Entr', 'problem_types': ['binary', 'multiclass']}},
        {'criterion': 'mse', 'n_estimators': 1000, 'ag_args': {'name_suffix': 'MSE', 'problem_types': ['regression']}},
    ],
    'KNN': [
        {'weights': 'uniform', 'ag_args': {'name_suffix': 'Unif'}},
        {'weights': 'distance', 'ag_args': {'name_suffix': 'Dist'}},
    ],
}