An issue about the default initialize methods.

Hello, I’m training some custom models with ESPNet2. And I find that the default initialize method checking the bias parameters of NN by the p.dim() function. This check is not precise (some parameters my also make p.dim() == 1 be True ) and set them to zero may lead to abnormal training. (I found one of my module always output zeros and the training may dead with ReLU. After removing the following lines, the training becomes OK. )

https://github.com/espnet/espnet/blob/96bd74641ceb463096067223d0734f70bddd8def/espnet2/torch_utils/initialize.py#L77-L80

[popcornell]

for name, p in model.named_parameters(): 
    if 'bias' in name:
        p.data.zero_() 

maybe make it case insensitive too just to be sure. This is difficult to implement in a scalable way. Current solution thus makes sense.

Maybe raising a warning also could be useful so user knows that the bias is set to zero. But warning should be likely raised once (e.g. for each layer of that class) otherwise will be too verbose maybe.

[kan-bayashi]

I found this issue has an impact on the most of TTS models since TTS modules is initialized using this method as a default. https://github.com/espnet/espnet/blob/14fcb2d42b2609f766ffaa7a79e9c921cd8398d9/espnet2/tts/fastspeech2/fastspeech2.py#L470-L475 https://github.com/espnet/espnet/blob/14fcb2d42b2609f766ffaa7a79e9c921cd8398d9/espnet2/tts/fastspeech2/fastspeech2.py#L828-L829

The model including BatchNorm2d or BatchNorm1d modules should be improved with this changes. (I’m not sure what is happened before fixing this issue…)