[Question] How to get the number of target algorithm runs without fitting data?

Hi there,

To get the number of target algorithm runs, auto-sklearn needs to fit data, and finally call sprint_statistics(). Is there any other way to get the number of target algorithm runs without fitting data? Since in my case, it just got 10 runs under time_left_for_this_wask=3600. To evaluate more runs. I know I could increase the time limit. In addition to do in this way, does anyone know how auto-sklearn decides the number of target algorithm runs?

Thanks a lot.

[eddiebergman]

Hi @jmren168,

We use SMAC3 as a Bayesian optimization (BO) framework. You can refer to the reference given there. Specifically, where we instansiate it is here.

I won’t go into details as BO is a huge field but the core principles:

• Every configuration x can be seen as a point in the very high-dimensional space X.
• At each point x, we would love to know its true performance p ahead of time but we only know it after evaluating it.
• A BO model tries to map each x -> u, s, a mean and std deviations, so it builds a model of the entire space X.
• With each configuration x evaluated, the BO model is updated and knows a little more about the whole landscape X. Essentially it’s a statistical model that learns to model the space X.
• There’s an exploration-exploitation tradeoff that happens to decide on next points to evaluate. Parts of the space X we don’t know much about will be tried once in a while but also to exploit regions of X where we do know a lot and shows good performance.

[jmren168]

Hi @eddiebergman,

After reading the paper you mentioned, there are 4 types of SMAC3, which one is really implemented in autosklearn? Some of them? All of them? Thanks.

SMAC4BB: SMAC for Low-dimensional and Continuous Black-Box Functions SMAC4HPO: SMAC for CASH and Structured Hyperparameter Optimization SMAC4MF: SMAC for Expensive Tasks and Automated Deep Learning SMAC4AC: SMAC for Algorithm Configuration