CoNLL2003 ner_tags order mismatch between the dataset from HF and the pretrained model

@patrickvonplaten @dslim23

@dslim23 's pretrained models such as: https://huggingface.co/dslim/bert-base-NER have the following NER tag order baked in: "O", "B-MISC", "I-MISC", "B-PER", "I-PER", "B-ORG", "I-ORG", "B-LOC", "I-LOC"

while the https://huggingface.co/datasets/conll2003 dataset has: O (0), B-PER (1), I-PER (2), B-ORG (3), I-ORG (4) B-LOC (5), I-LOC (6) B-MISC (7), I-MISC (8)

The mismatch leads to defunct accuracy measurements out of the box for the pretrained NER models; try, for instance: python examples/pytorch/token-classification/run_ner.py --model_name_or_path dslim/bert-base-NER --dataset_name conll2003 --output_dir /tmp/test-ner --do_eval

[julien-c]

@patrickvonplaten thanks for the ping – though in the case it’s the script that should be able to remap labels no? The model looks correctly defined with https://huggingface.co/dslim/bert-base-NER/blob/main/config.json

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.