Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

`compute_metrics` show better results than `generate` because target data leaks

See original GitHub issue

Environment info

transformers version: 4.3.2
Platform: Linux-5.8.18-050818-generic-x86_64-with-glibc2.10
Python version: 3.8.5
PyTorch version (GPU?): 1.7.1 (True)
Tensorflow version (GPU?): not installed (NA)
Using GPU in script?: yes
Using distributed or parallel set-up in script?: no

Who can help

Information

Model I am using:

The problem arises when using:

my own modified scripts: (give details below)

The tasks I am working on is:

my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

train model with compute_metrics function to monitor metrics
use generate to predict text with trained model

Expected behavior

I expect the metrics of compute_metrics to be equal to my generated text.

More information

While training, I used compute_metrics to calculate the metric on my validation set every X steps. I was surprised to see that after training my model did not perform as expected using the generate function provided by huggingface.

After some digging through the code I think I understand what the problem is. compute_metrics takes as input preds, which is a collection of logits from prediction_step which internally calls model with the inputs and targets of the model. This means that the target text leaks into preds.predictions because mode.forward used the targets as input for the decoder. This makes the metrics of compute_metrics seem much better than they really are.

In my opinion the target data should not be used to create preds.predictions. Maybe the generate function is a better fit.

Issue Analytics

State:
Created 3 years ago
Reactions:2
Comments:5 (3 by maintainers)

Top GitHub Comments

2reactions

sguggercommented, Feb 18, 2021

Did you use the flag --predict_with_generate? It’s there just for this: predicting using the generate method and the labels are then not passed (except to compute the loss).

0reactions

github-actions[bot]commented, Apr 14, 2021

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

Top Results From Across the Web

How to Avoid Data Leakage When Performing Data Preparation

Naive application of data preparation methods to the whole dataset results in data leakage that causes incorrect estimates of model performance.

Be Careful from Data Leakage in your Machine Learning Model

Data leakage refers to a mistake that is made by the creator of a machine learning model in which information about the target...

Data Leakage in Machine Learning: How it can be detected ...

We can define data leakage as: “When data set contains relevant data, but similar data is not obtainable when the models are used...

Best practices for creating training data | AutoML Tables

Target leakage can cause your model to show excellent evaluation metrics, but perform poorly on real data.

10. Common pitfalls and recommended practices - Scikit-learn

10.2.1. Data leakage during pre-processing¶ · Using all the data to perform feature selection results in an accuracy score much higher than chance,...