rouge-score not found by evaluate.load()

Hello, I have a problem loading the rouge metric with evaluate.load(). Apparently, the rouge_scores library (https://github.com/google-research/google-research/tree/master/rouge) is not found by evaluate. I followed your tutorial from https://huggingface.co/course/chapter7/5?fw=pt#preprocessing-the-data (section ‘Metrics for Text Summarization’) and installed the rouge_score library. I got access to two different clusters. On the first one, everything works like a charm, however, the second cluster throws this exception below:

Traceback

Traceback (most recent call last): File “/home/hd/hd_hd/hd_rk435/MA_Thesis/fine-tune/mnli_text/generation_MNLI.py”, line 14, in <module> rouge_score = evaluate.load(“rouge”) #/home/hd/hd_hd/hd_rk435/rouge") # evaluate.load(“rouge”) #/pfs/data5/home/hd/hd_hd/hd_rk435/ma_venv/lib64/python3.8/site-packages/rouge_score/rouge_scorer.py") #“rouge”) File “/pfs/data5/home/hd/hd_hd/hd_rk435/ma_venv/lib64/python3.8/site-packages/evaluate/loading.py”, line 716, in load evaluation_module = evaluation_module_factory( File “/pfs/data5/home/hd/hd_hd/hd_rk435/ma_venv/lib64/python3.8/site-packages/evaluate/loading.py”, line 621, in evaluation_module_factory return LocalEvaluationModuleFactory( File “/pfs/data5/home/hd/hd_hd/hd_rk435/ma_venv/lib64/python3.8/site-packages/evaluate/loading.py”, line 424, in get_module local_imports = _download_additional_modules( File “/pfs/data5/home/hd/hd_hd/hd_rk435/ma_venv/lib64/python3.8/site-packages/evaluate/loading.py”, line 268, in _download_additional_modules raise ImportError( ImportError: To be able to use rouge, you need to install the following dependencies[‘rouge’] using ‘pip install rouge’ for instance’

Also for Python 3.9

Traceback (most recent call last): File “/home/hd/hd_hd/hd_rk435/MA_Thesis/fine-tune/mnli_text/generation_MNLI.py”, line 14, in <module> rouge_score = evaluate.load(“/home/hd/hd_hd/hd_rk435/rouge/rouge.py”) # evaluate.load(“rouge”) #/pfs/data5/home/hd/hd_hd/hd_rk435/ma_venv/lib64/python3.8/site-packages/rouge_score/rouge_scorer.py") #“rouge”) File “/pfs/data5/home/hd/hd_hd/hd_rk435/test_venv/lib64/python3.9/site-packages/evaluate/loading.py”, line 774, in load evaluation_module = evaluation_module_factory( File “/pfs/data5/home/hd/hd_hd/hd_rk435/test_venv/lib64/python3.9/site-packages/evaluate/loading.py”, line 671, in evaluation_module_factory return LocalEvaluationModuleFactory( File “/pfs/data5/home/hd/hd_hd/hd_rk435/test_venv/lib64/python3.9/site-packages/evaluate/loading.py”, line 494, in get_module local_imports = _download_additional_modules( File “/pfs/data5/home/hd/hd_hd/hd_rk435/test_venv/lib64/python3.9/site-packages/evaluate/loading.py”, line 269, in _download_additional_modules raise ImportError( ImportError: To be able to use rouge, you need to install the following dependencies[‘rouge’, ‘rouge’, ‘rouge’] using ‘pip install rouge rouge rouge’ for instance’

Code to reproduce:

import evaluate
rouge_score = evaluate.load("rouge")

Different ways of loading the metric, e.g through a path results in the same exception.

Versions I’ve tried to get the script running in different versions, none of them worked.

• Python 3.8.6/3.9.2
• transformers 4.19.2
• rouge-score 1.0.2/0.0.4
• evaluate 0.3.0/0.1.2
• Torch 1.12.1

On the first cluster I’m using the following versions:

• Python 3.8.10
• transformers 4.19.2
• rouge-score 0.0.4
• evaluate 0.1.2
• torch 1.12.0

Rouge Score seems to be correctly installed:

(test_venv) [ ~]$ ls test_venv/lib64/python3.9/site-packages/rouge_score/
__init__.py  create_pyrouge_files.py  io_test.py  rouge_scorer.py       scoring.py       test_util.py  tokenize_test.py
__pycache__  io.py                    rouge.py    rouge_scorer_test.py  scoring_test.py  tokenize.py

Thanks for reading!

[lvwerra]

I think loading rouge_score before might lead to a namespace clash with the evaluate importer in the last example.

What’s odd in the original message is that it says you should install rouge whereas the import is actually rouge_score. The error comes from here: https://github.com/huggingface/evaluate/blob/960dc92ef6a86e3b11d7b7276b4960135a3129b9/src/evaluate/loading.py#L262-L272

When looking at the library_import_name, and library_import_path I get:

absl # Here to have a nice missing dependency error message early on
datasets datasets
nltk # Here to have a nice missing dependency error message early on
numpy # Here to have a nice missing dependency error message early on
six # Here to have a nice missing dependency error message early on
rouge_score rouge_score
evaluate evaluate

So it is not clear to me where rouge comes from since this is grabbed by scanning the Python files for imports.

Anyway, since it is fixed now I won’t spend more time investigating 😃

[PhMeier]

I’ve used version 0.3.0 of evaluate and rouge-score 1.2.1 in Python 3.9.2 with this code example:

from rouge_score import rouge_scorer, scoring
import evaluate
from transformers import AutoTokenizer, BartTokenizer

rouge_score = evaluate.load("/home/hd/hd_hd/hd_rk435/evaluate/metrics/rouge")
tokenizer = BartTokenizer.from_pretrained("facebook/bart-large")
generated_summary = "Nearly all the tips useful and. I consider myself an intermediate to advanced user of OneNote."
reference_summary = "Review: Nearly all the tips useful and. I consider myself an intermediate to advanced user of OneNote."

res = rouge_score.compute(predictions=[generated_summary], references=[reference_summary], use_stemmer=True) #, tokenizer=tokenizer)
print(res) 

{'rouge1': 0.9696969696969697, 'rouge2': 0.967741935483871, 'rougeL': 0.9696969696969697, 'rougeLsum': 0.9696969696969697}

Now it works! I’m pretty sure that I’ve tried this combination yesterday.

Anyways thanks again for the fast reply and this great library!