bart-large-cnn ROUGE-L scores

Environment info

Who can help

BART + Summarization @sshleifer

Information

Model I am using is BART.

The problem arises when: verifying accuracy numbers of facebook/bart-large-cnn on CNN+Daily Mail. The paper reports R1, R2, RL of 44.16, 21.28, 40.90 but I can get only 44.05, 21.07, 30.62. I used this to make my dataset. Is this expected?

The tasks I am working on is:

• CNN-Dm summarization task

To reproduce

Steps to reproduce the behavior:

1. Follow instructions to download dataset
2. Run with python run_summarization.py --reference_path=data/cnn_dm/test.target data/cnn_dm/test.source results/test.log

[yxyzzz]

1. rouge_score split sentences by ‘\n’. You can add a ‘\n’ to separate sentences in the summaries and evaluate. The summary level rougeL (rougeLsum) should be a lot higher and close to the one in the literature. ‘{‘rouge1’: 44.0536, ‘rouge2’: 21.0711, ‘rougeL’: 30.6157, ‘rougeLsum’: 40.9812}’

output_ln2 = []
for o in `output_ln:
    s = sent_tokenize(p)
    output_ln2.append('\n'.join(s))

1. Use pyrouge -> https://pypi.org/project/pyrouge/

[swethmandava]

Opened a PR at #7356 that fixes this issue @sshleifer