How to implement transfer learning / adaptation for tts

Now I try to implement speaker adaptation toward pre-trained model. I added torch_load(args.model, model) into espnet/tts/pytorch_backend/tts.py. (Codes) But the loss became the same outline even if transformer-lr decreased. (Results) * encoder_alpha & decoder_alpha changed by decreasing transformer-lr.

Could you give me some advise? e.g., adding some codes, changing some configs, …

[Adaptation condition] Pre-trained model: libritts.transformer.v1 Adaptation data: test_clean (open speaker for training) Adaptation config:

• transformer-lr: 1 -> transformer-lr: 1e-8
• epochs: 100 -> epochs: 2
• others becomes the same setting

[key wards] transfer learning, speaker adaptation, fine-tuning

[kan-bayashi]

And I heard that libritts includes pose in speech without any corresponding text. This causes training failure. To avoid this issue, VAE-based tacotron is needed. https://arxiv.org/pdf/1810.07217.pdf

If I have a time, I will try to implement this model.

[potato-inoue]

I checked that single-speaker model adaptation is running. model.eval() was the reason to not train. I have no pre-process for pose above you mentioned. Multi-speaker model has not checked yet.

• model: ljspeech.transformer.v1
• adapted speaker: speaker 237 (Female) from test_clean