Finetune Hubert model : Adding new vocabulary

Environment info

transformers version: 4.12.2
Platform: Mac
Python version: 3.7
PyTorch version (GPU?): 1.9
Tensorflow version (GPU?): No
Using GPU in script?: No
Using distributed or parallel setup in script?: No

I just run simple code to load Hubert pretrained base model

from transformers import Wav2Vec2Processor, HubertForCTC
import torch
import librosa
PROCESSOR = Wav2Vec2Processor.from_pretrained('facebook/hubert-large-ls960-ft')
model = HubertForCTC.from_pretrained('facebook/hubert-large-ls960-ft')
tokenizer = PROCESSOR.tokenizer

On a smaller dataset, i am able to get good WER around 0.0

But if I add new tokens/vocabulary to it by using the below code:

from transformers import Wav2Vec2Processor, HubertForCTC
import torch
import librosa
PROCESSOR = Wav2Vec2Processor.from_pretrained('facebook/hubert-large-ls960-ft')
model = HubertForCTC.from_pretrained('facebook/hubert-large-ls960-ft')
tokenizer = PROCESSOR.tokenizer
tokenizer.add_tokens(new_tokens=[' ','Ä','Ö','Ü'])

The loss and WER go bad and then worse (clearly), and later loss is NAN.

Is it the correct way to add new alphabets?

dataset is same in both trainings

[harrypotter90]

Cool, it worked. Thank you

[patrickvonplaten]

Hey @harrypotter90,

exactly sorry I forgot to mention this parameter. To summarize, I would recommend to add new tokens and load your model as follows:

from transformers import Wav2Vec2Processor, AutoModelForCTC

# load tokenizer & feature extractor
processor = Wav2Vec2Processor.from_pretrained('facebook/hubert-large-ls960-ft')

# add new tokens
tokenizer = processor.tokenizer
tokenizer.add_tokens(new_tokens=['Ä','Ö','Ü'])

# load pretrained model and replace fine-tuned head with resized randomly initialized head
model = AutoModelForCTC.from_pretrained("facebook/hubert-large-ls960-ft", vocab_size=len(tokenizer), ignore_mismatched_sizes=True)

# now use model for training