Inflexible score function of CheckpointSaver

https://github.com/Project-MONAI/MONAI/blob/9f9b63a7dc56d8b31d4fc8d34aae77865ba95947/monai/handlers/checkpoint_saver.py#L148-L160

Currently the CheckpointSaver saves the check point that has the largest score value. However, this is not desirable when a lower score indicates better performance, e.g. validation loss. We can either allow the user to pass a custom score function (as in the ignite.handlers), or add a flag to control returning round(engine.state.metrics[metric_name], 4) or - round(engine.state.metrics[metric_name], 4)

[vfdev-5]

Hi @Nic-Ma , in ignite we provide (since v0.4.3) a default score function from metric : https://pytorch.org/ignite/generated/ignite.handlers.checkpoint.Checkpoint.html#ignite.handlers.checkpoint.Checkpoint.get_default_score_fn in addition to score_function argument such that both cases: larger-better or lower-better are covered:

from ignite.handlers import Checkpoint

best_acc_score = Checkpoint.get_default_score_fn("accuracy")
best_model_handler = Checkpoint(
    to_save, save_handler, score_name="val_accuracy", score_function=best_acc_score
)
evaluator.add_event_handler(Events.COMPLETED, best_model_handler)

and

from ignite.handlers import Checkpoint

neg_loss_score = Checkpoint.get_default_score_fn("loss", -1.0)
best_model_handler = Checkpoint(
    to_save, save_handler, score_name="val_neg_loss", score_function=neg_loss_score
)
evaluator.add_event_handler(Events.COMPLETED, best_model_handler)

Maybe, something similar could be done from MONAI’s side ?

[vfdev-5]

@Nic-Ma thanks ! Yes, I agree that it is better to show actual value instead of negative one. We had previously abs(score) such that negative values become positive but we removed that. Now, the main pain point is that score function return negative value as well…