Evaluating BART on CNN/DM : How to process dataset

From the README of BART for reproducing CNN/DM results :

Follow instructions here to download and process into data-files such that test.source and test.target has one line for each non-tokenized sample.

After following instructions, I don’t have files like test.source and test.target…

Instead, I have test.bin, and chunked version of this file
(chunked/test_000.bin ~ chunked/test_011.bin).

---

How can I process test.bin into test.source and test.target ?

@ngoyal2707 @yinhanliu

[artmatsak]

Here’s a version for Python 3 if anyone is interested:

https://github.com/artmatsak/cnn-dailymail

[zhaoguangxiang]

There are many details, here is my code.

I fix the over lenght of train.bpe.source caused by ascii ‘0D’ in articles by split and join

I summarize several notes here :

1. remove " " before “.”
2. cased, remove the line of lower cased
3. “\r” in origin articles leads error in bpe preprocess
4. remove “(CNN)”
5. bpe encoding

code : https://gist.github.com/zhaoguangxiang/45bf39c528cf7fb7853bffba7fe57c7e