Question about LayerNorm in Wav2vec

Hi @TParcollet ,

I have a question about the layer normalization used in wav2vec for both HF and Fairseq implementations. Why using it on all (batch_size, sequence_length, hidden_size) dimensions of the wav2vec last_hidden_state output instead of just on the hidden_size dimension, as showed in the examples of pytorch doc here (maybe better call it InstanceNorm).

Thanks in advance !

https://github.com/speechbrain/speechbrain/blob/424e7921531b0ea6523557ef0fd6ca249936bd26/speechbrain/lobes/models/huggingface_wav2vec.py#L280-L285

https://github.com/speechbrain/speechbrain/blob/424e7921531b0ea6523557ef0fd6ca249936bd26/speechbrain/lobes/models/fairseq_wav2vec.py#L172-L176

[TParcollet]

1. It can be done in the dataloader. But I already noted that we should add our padded-compatible layernorm on these recipes. Honestly, I wouldn’t expect this to impact to much the results … It’s adding regularization.

[bofenghuang]

Hi @TParcollet ,

I think I’ve made some mistakes here.

1. Normalizing each wav is ok, but it should not include the padded values. I’ve forgot the wav is a padded batch… It should better be done somewhere when making dataloader or pass wav_lens to model and do it after.

2. Layer normalization has already been done in HF and fairseq (with trained w and b). Should do it one more time here ?