default value for default_label_names incorrectly causes has_label in trainer.py to be true

System Info

• transformers version: 4.24.0
• Platform: Linux-4.18.0-372.26.1.el8_6.x86_64-x86_64-with-glibc2.17
• Python version: 3.8.13
• Huggingface_hub version: 0.10.1
• PyTorch version (GPU?): 1.11.0+cu113 (False)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed

Who can help?

@sgugger This is a bug in trainer.py

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, …)
• My own task or dataset (give details below)

Reproduction

I’ve upgraded transformers from 4.17.0 to 4.24.0. In the earlier version(s), the default value for default_label_names was set to ["labels"]. In the current version the default is set to an empty array []

4.17.0:

default_label_names = (
            ["start_positions", "end_positions"]
            if type(self.model).__name__ in MODEL_FOR_QUESTION_ANSWERING_MAPPING_NAMES.values()
            else ["labels"]
        )

4.24.0:

default_label_names = find_labels(self.model.__class__)

default_label_names is used to set self.label_names if no labels are supplied:

self.label_names = default_label_names if self.args.label_names is None else self.args.label_names

It will then be used to decide the value of has_labels as True or False:

        has_labels = all(inputs.get(k) is not None for k in self.label_names)

In 4.17.0 with the default value of ["labels"] has_labels was False. In 4.24.0 with the default value of [] has_labels is True causing the program to fail in compute_loss during eval:

E               ValueError: The model did not return a loss from the inputs, only the following keys: start_logits,end_logits,target_type_logits. For reference, the inputs it received are input_ids,attention_mask.

Expected behavior

The default behavior when no labels are provided should cause has_labels to be False

[seirasto]

This worked! Thank you!!

[sgugger]

Ah yes, I understand your concern better, thanks! Will adapt the PR.