Resume study with new metric

(Not sure if the feature has implemented or not) When we have multiple metric that can evaluate the models, e.g., metric A and B. Assume we have trained 100 trials for metric A and saved the prediction of 100 models. If we want another study to optimize metric B, then compare to begin a new study, we can easily compute last 100 trials’ score of metric B as a better prior.

Motivation

Now I am tuning a model that predict financial market, there are lots of metrics to evaluate the models(for example hit ratio, sharpe ratio, average pl…). How the parameters differ from metrics is interesting. Restart the study of new metric with the prior using the trained trials can save a lot of time.

Description

Simplest way is just give an array of metric_B_value = (number, value) which is tried trials’ score of new metric. Then start a new study base on this array and the study history with metric A. For example: study = optuna.create_study(study_name='Study_metric_B', storage='sqlite:///example.db', study_inherit = "Study_metric_A", new_metric =metric_B_value)

[sile]

Thank you for creating this issue. Your motivation is interesting. Although the current Optuna doesn’t provide a feature that directly satisfies your requirement, I think that a private method Study._append_trial() can be used for such purpose (see the following toy example):

import optuna

def objective(trial, metric_name):
    x = trial.suggest_uniform('x', -10 , 10)
    y = trial.suggest_uniform('y', 3, 30)

    metrics = {
        'plus': x + y,
        'minus': x - y
    }
    trial.set_user_attr('metrics', metrics)
   
    return metrics[metric_name]    

# Run the first optimization that uses 'plus' metric.
study_a  = optuna.create_study()
study_a.optimize(lambda trial: objective(trial, 'plus'), n_trials=100)

# Import the result of `study_a`
study_b = optuna.create_study()
for trial in study_a.trials:
    study_b._append_trial(
        value=trial.user_attrs['metrics']['minus'],  # Change the metric to 'minus'
        params=trial.params,
        distributions=trial.distributions,
        user_attrs=trial.user_attrs,
        system_attrs=trial.system_attrs
   )
assert len(study_b.trials) == 100

# Run the second optimization that uses 'minus' metric.
study_b.optimize(lambda trial: objective(trial, 'minus'), n_trials=100)
assert len(study_b.trials) == 200

What do you think of it?

[sile]

Since this issue itself seems to have been solved and there is a related issue #821, please let me close it. FYI, we have implemented the multi-objective feature by #1054.