Fine-tune optimal hyperparameters for small datasets?

I’m trying to improve the accuracy of the ASR model for a specific domain.

The base model it’s a fine-tuned model from English (QuartzNet) to my language, and now I’m trying to improve this base model on a specific dataset.

The dataset has around 12h, and I used speed perturbation to generate around 37h.

I use these hyperparameters by the loss seems to stay the same after a significant number of epochs.

new_optimizer = {
        'name': 'novograd',
        'lr': 0.0001,
        'betas': (0.95, 0.25),
        'weight_decay': 0.001,
        'sched': {
            'last_epoch': -1,
            'min_lr': 0.0,
            'monitor': 'val_loss',
            'name': 'CosineAnnealing',
            'reduce_on_plateau': False,
            'warmup_ratio': 0.12,
            'warmup_steps': None,
        },
    }

I use 500 epochs for this training.

Do you have any advice on how I can optimally improve the accuracy of this small dataset, or how is a minimum number of hours that can generate a good improvement?

[titu1994]

If you dataset is so small, it’s best to freeze the encoder then unfreeze the batch norm. Irrespective of same or different language.

[adiIspas]

Another short question @titu1994.

Batch normalization has to been unfrozen if the language remains the same, and I perform some specific domain fine-tune?