wav2vec pretrain help

❓ Questions and Help

i’m trying to pretrain custom wav2vec2 model using my own dataset. The dataset size is about 10k hour. The official wav2vec2 base model was used as parameter initialization. The training loss suddenly drop a lot after a few epoch training and validation loss become higher.

Before asking:

1. search the issues.
2. search the docs.

What is your question?

1. Training loss (purple )doesn’t look right. Validation loss (red) become higher

2. does code perplexity curve look normal?

3. does gradient curve look normal?

@alexeib can you kindly help? Thanks.

Code

use the same config as wav2vec2 base model.

What have you tried?

i tried lower the learning rate and fp32 training instead of fp16, but doesn’t help.

What’s your environment?

• fairseq Version (e.g., 1.0 or master): master
• PyTorch Version (e.g., 1.0)1.7.1
• OS (e.g., Linux): Linnux
• How you installed fairseq (pip, source): source
• Build command you used (if compiling from source):pip install -e
• Python version:3.7
• CUDA/cuDNN version:11.0
• GPU models and configuration: 4 V100
• Any other relevant information:

[alexeib]

extractor_mode: layer_norm is much more stable and typically has similar performance to default (just make sure you set feature_grad_mult to 1.0 and task.normalize=true)

layer_norm_first allows you to train beyond 500k updates without crashing in fp16 mode. by itself it is not as accurate as post layer norm, but when you train for longer you outperform post layer norm models. for this to be effective you you need to significantly increase the learning rate as compared to post layer norm model (by 20-30x)

[stale[bot]]

Closing this issue after a prolonged period of inactivity. If this issue is still present in the latest release, please create a new issue with up-to-date information. Thank you!