state.best_metric does not update in EarlyStoppingCallback

https://github.com/huggingface/transformers/blob/b18dfd95e1f60ae65a959a7b255fc06522170d1b/src/transformers/trainer_callback.py#L534

I print the state.best_metric, but it is always None. I wonder whether the state.best_metric should be updated if the condition is satisfied, like the following:

if state.best_metric is None or (
            operator(metric_value, state.best_metric)
            and abs(metric_value - state.best_metric) > self.early_stopping_threshold
        ):
            self.early_stopping_patience_counter = 0
            state.best_metric = metric_value

[baskrahmer]

Cool, I’ll be working on one!

[datalee]

I have set load_best_model_at_end to True, but it doesn’t work. The following is my code:

args_list=[...]
parser = HfArgumentParser((ModelArguments, DataTrainingArguments, Seq2SeqTrainingArguments))
model_args, data_args, training_args = parser.parse_args_into_dataclasses(args_list)
training_args.load_best_model_at_end = True
training_args.metric_for_best_model = 'eval_rouge-l'
training_args.greater_is_better = True

However, when I add control.should_save in

https://github.com/huggingface/transformers/blob/b18dfd95e1f60ae65a959a7b255fc06522170d1b/src/transformers/trainer_callback.py#L531

it works. Just like this:

def check_metric_value(self, args, state, control, metric_value):
      # best_metric is set by code for load_best_model
      operator = np.greater if args.greater_is_better else np.less
      if state.best_metric is None or (
          operator(metric_value, state.best_metric)
          and abs(metric_value - state.best_metric) > self.early_stopping_threshold
      ):
          self.early_stopping_patience_counter = 0
          control.should_save = True
      else:
          self.early_stopping_patience_counter += 1
          control.should_save = False

not work for me