How to train a custom seq2seq model with BertModel

How to train a custom seq2seq model with BertModel,

I would like to use some Chinese pretrained model base on BertModel

so I’ve tried using Encoder-Decoder Model, but it seems theEncoder-Decoder Model is not used for conditional text generation

and I saw that BartModel seems to be the model I need, but I cannot load pretrained BertModel weight with BartModel.

by the way, could I finetune a BartModel for seq2seq with custom data ?

any suggestion, thanks

[patrickvonplaten]

Hi @chenjunweii - thanks for your issue! I will take a deeper look at the EncoderDecoder framework at the end of this week and should add a google colab on how to fine-tune it.

[patrickvonplaten]

Yeah, the code is ready in this PR: https://github.com/huggingface/transformers/tree/more_general_trainer_metric . The script to train an Encoder-Decoder model can be assessed here: https://github.com/huggingface/transformers/blob/more_general_trainer_metric/src/transformers/bert_encoder_decoder_summary.py

And in order for the script to work, you need to use this Trainer class: https://github.com/huggingface/transformers/blob/more_general_trainer_metric/src/transformers/trainer.py

I’m currently training the model myself. When the results are decent, I will publish a little notebook.