Mode for key_metric

Is your feature request related to a problem? Please describe. I think the Workflow or CheckpointSaver need a mode option for key_metric and additional_metrics to handle both metrics like acc and mse.

Describe the solution you’d like Here is the keras version:

if mode == 'min':
      self.monitor_op = np.less
      self.best = np.Inf
elif mode == 'max':
      self.monitor_op = np.greater
      self.best = -np.Inf
else: #auto mode
      if 'acc' in self.monitor or self.monitor.startswith('fmeasure'):
        self.monitor_op = np.greater
        self.best = -np.Inf
      else:
        self.monitor_op = np.less
        self.best = np.Inf

[vfdev-5]

Hi @Nic-Ma here is an example of usage of ignite’s Checkpoint with score_function to adapt mse as “increasing score” :

from ignite.engine import Engine, Events
from ignite.handlers import Checkpoint, DiskSaver, global_step_from_engine

trainer = ...
evaluator = ...

def score_function(engine):
    return -1.0 * engine.state.metrics['mse']

to_save = {'model': model}
handler = Checkpoint(
    to_save, 
    DiskSaver('/tmp/models', create_dir=True), 
    n_saved=2,
    filename_prefix='best', 
    score_function=score_function, 
    score_name="neg_val_mse",
    global_step_transform=global_step_from_engine(trainer)
)
evaluator.add_event_handler(Events.COMPLETED, handler)

> ["best_model_9_neg_val_mse=-0.15.pt", "best_model_10_neg_val_mse=-0.08.pt", ]

HTH

[vfdev-5]

As far as I understand, the feature request is to provide 2 + “auto mode” ways to compare metrics to decide what to save. In ignite we assume to save best score as larger score and for metrics like MSE (where better is lower vs Accuracy where better is greater) we ask user to provide inverse of it such that the best model has larger score.

I think auto magic mode it is better to avoid as could lead to unexpected errors or user misunderstanding of the usage.