Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

default value for default_label_names incorrectly causes has_label in to be true

See original GitHub issue

System Info

  • transformers version: 4.24.0
  • Platform: Linux-4.18.0-372.26.1.el8_6.x86_64-x86_64-with-glibc2.17
  • Python version: 3.8.13
  • Huggingface_hub version: 0.10.1
  • PyTorch version (GPU?): 1.11.0+cu113 (False)
  • Tensorflow version (GPU?): not installed (NA)
  • Flax version (CPU?/GPU?/TPU?): not installed (NA)
  • Jax version: not installed
  • JaxLib version: not installed

Who can help?

@sgugger This is a bug in


  • The official example scripts
  • My own modified scripts


  • An officially supported task in the examples folder (such as GLUE/SQuAD, …)
  • My own task or dataset (give details below)


I’ve upgraded transformers from 4.17.0 to 4.24.0. In the earlier version(s), the default value for default_label_names was set to ["labels"]. In the current version the default is set to an empty array []


default_label_names = (
            ["start_positions", "end_positions"]
            if type(self.model).__name__ in MODEL_FOR_QUESTION_ANSWERING_MAPPING_NAMES.values()
            else ["labels"]


default_label_names = find_labels(self.model.__class__)

default_label_names is used to set self.label_names if no labels are supplied:

self.label_names = default_label_names if self.args.label_names is None else self.args.label_names

It will then be used to decide the value of has_labels as True or False:

        has_labels = all(inputs.get(k) is not None for k in self.label_names)

In 4.17.0 with the default value of ["labels"] has_labels was False. In 4.24.0 with the default value of [] has_labels is True causing the program to fail in compute_loss during eval:

E               ValueError: The model did not return a loss from the inputs, only the following keys: start_logits,end_logits,target_type_logits. For reference, the inputs it received are input_ids,attention_mask.

Expected behavior

The default behavior when no labels are provided should cause has_labels to be False

Issue Analytics

  • State:closed
  • Created 10 months ago
  • Comments:13 (7 by maintainers)

github_iconTop GitHub Comments

seirastocommented, Nov 8, 2022

This worked! Thank you!!

sguggercommented, Nov 8, 2022

Ah yes, I understand your concern better, thanks! Will adapt the PR.

Read more comments on GitHub >

github_iconTop Results From Across the Web

The default value of `cuda_device` of the class ... - GitHub
I think the reason is that in the class Trainer , it set the cuda_device to cuda:0 if the cuda_device is None and...
Read more >
Trainer - Hugging Face
The Trainer class provides an API for feature-complete training in PyTorch for most standard use cases. It's used in most of the example...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found