Why change sequence order of prev_output_tokens in BART?

Hi,

In the extract_features example of BART (link to code), both src_tokens (for encoder) and prev_output_tokens (for decoder) are fed into model, but prev_output_tokens is just the src_tokens with EOS being moved to the beginning. I think during training (teacher forcing) the prev_output_tokens starts with BOS, doesn’t it? So what’s the purpose of this reorder? To add some deliberate noise, or assuming there’s another sentence before it?

Thanks in advance.

Rui

[ngoyal2707]

https://github.com/pytorch/fairseq/blob/master/fairseq/data/language_pair_dataset.py#L63 is the fixed link. Ahh I see, so actually the input format to encoder for above case is actually:

<s> A <mask> B <mask> E </s>

and Decoder input:

</s> <s> A B C D E

And target:

<s> A B C D E </s>

We will look into updating those figures. Thanks for pointing it out.

[yinhanliu]

@Colanim @villmow

1. our release cnn/dm model was fine-tuned with bos. But you can totally remove bos during fine-tune and it works as well. The only thing you want to make sure is that if you don’t have bos in fine-tune then don’t use it in test, if you do use in fine-tune, then keep it in test.
2. Pretrain doesn’t need bos at all. Bos was a concept from Roberta, and turned out not necessary at all.