Issue with HANS evaluation

🐛 Bug

Information

Model I am using (Bert, XLNet …): Albert, Bert

Language I am using the model on (English, Chinese …): GLUE (MNLI), HANS

The problem arises when using:

• the official example scripts: (give details below)
• my own modified scripts: (give details below)

The tasks I am working on is:

• an official GLUE/SQUaD task: (give the name)
• my own task or dataset: (give details below)

To reproduce

I have tried to evaluate ALBERT and BERT (pretrained, trained on MNLI) on HANS. I’m using the official example.

Everytime I run evaluation, I’m getting

Heuristic entailed results:
lexical_overlap: 0.0
subsequence: 0.0
constituent: 0.0

Heuristic non-entailed results:
lexical_overlap: 1.0
subsequence: 1.0
constituent: 1.0

Results can be reproduced by:

python3 hans/test_hans.py --task_name hans --model_type bert/albert --do_eval --data_dir $HANS_DIR --model_name_or_path $MODEL_PATH --max_seq_length 128 --output_dir $MODEL_PATH --per_gpu_eval_batch_size 1024 --overwrite_cache

Expected behavior

It shouldn’t be exactly 0 and 1 always and entailment score should be higher than non entailment.

Environment info

• transformers version: 2.11.0
• Platform: Linux 5.4.0-29-generic
• Python version: 3.8.2
• PyTorch version (GPU?): 1.5.0+cu101
• Tensorflow version (GPU?): NA
• Using GPU in script?: Yes
• Using distributed or parallel set-up in script?: No

[sgugger]

I forgot the header on that file, #5082 should fix.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.