should mBART-large-en-ro have decoder_start_token_id by default?

Hypothesis: since the argument prepend_bos is set to “False” in fairseq/examples/README.md, mbart-large-en-ro does not need decoder_start_token_id.

TODO:

• create branch that deletes decoder_start_token_id. Setting it to None in the config might not be enough.
• verify that decoder_start_token_id is in fact not being used by setting a breakpoint in generate.
• run_eval.py on wmt-en-ro/test and see if BLEU is >= 26.46, the score with decoder_start_token_id=250020.

[sshleifer]

Gunna close this since the score is now basically the same as fairseq. Thanks for your help!

[sshleifer]

I think I fixed this another way in #6526 on master

python run_eval.py facebook/mbart-large-en-ro $ENRO_DIR/test.source eos_baseline_enro_test_generations.txt \
--reference_path $ENRO_DIR/test.target \
--score_path baseline_test_bleu_eos.json --bs 32 --task translation --fp16

=> {‘bleu’: 26.81}

python run_eval.py facebook/mbart-large-en-ro $ENRO_DIR/test.source \
eos_baseline_enro_test_generations.txt --reference_path $ENRO_DIR/test.target \
--score_path baseline_test_bleu_eos.json --bs 32 --task translation --fp16  \
--decoder_start_token_id 2

{‘bleu’: 11.57} (and takes 40 mins!)

in the original fairseq I get 26.83.