Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

should PegasusTokenizer replace `/n` with `<n>`?

See original GitHub issue

On the encode/text -> ids side I’m certain. On the decode/ids -> text side, I’m worried about breaking run_eval.py, which reads the generations from disk before calculating rouge here

cc @stas00

Issue Analytics

State:
Created 3 years ago
Comments:10 (10 by maintainers)

Top GitHub Comments

1reaction

sshleifercommented, Oct 17, 2020

(1) just pegasus (2) It doesn’t handle the situation – it leaves <n> in the output and trusts calculate_rouge_score (which calls add_newline_to_end_of_each_sentence) to temporarily remove <n> and then add \n between sentences, thereby computing rougeLsum correctly. This happens after results are saved, and therefore generations still have <n>. https://github.com/huggingface/transformers/blob/dc552b9b7025ea9c38717f30ad3d69c2a972049d/examples/seq2seq/sentence_splitter.py#L18

1reaction

julien-ccommented, Oct 13, 2020

Your first line is composed of the characters \ and n and not the actual character customarily represented by \n which is hex(0a) (ascii code = 10), no?

Top Results From Across the Web

Pegasus - Hugging Face

Construct a “fast” PEGASUS tokenizer (backed by HuggingFace's tokenizers library). Based on Unigram. This tokenizer inherits from PreTrainedTokenizerFast which ...

NLP Data Augmentation

Synonym Replacement: Randomly choose n words from the sentence that does not stop words. Replace each of these words with one of its...

Quick Tip: How to capture and replace newlines in liquid for ...

Turns out, the main trick was to capture a newline in liquid and assemble the string to replace using that one. {% capture...

A Survey of Recent Abstract Summarization Techniques

We will explore its implementation on T5, Pegasus, and ProphetNet. ... ProphetNet recovers the next n future tokens within each masked token span....

Paper_summary.ipynb - Colaboratory - Google Colab

Off | 00000000:00:04.0 Off | 0 | | N/A 33C P0 23W / 300W | 0MiB / 16160MiB | 0% ... tokenizer =...