loss increasing with larger input range

When I multiply the input by sqrt(512), loss always increases, on the order of magnitude of 1e2~1e3 from the beginning of training. i.e., Only change fconv.py:

def Embedding(num_embeddings, embedding_dim, padding_idx):
    m = nn.Embedding(num_embeddings, embedding_dim, padding_idx=padding_idx)
    m.weight.data.normal_(0, 0.1)
    m.weight.data.mul_(math.sqrt(embedding_dim))  # I add it here
    return m

I thought the operation above only change the input’s magnitude from 1e-4 to 1e-2, can you tell me why loss exploding? PS: I’m sure loss decreases normally without mul_.

---

Here is part of the log:

| epoch 001:   0%|                                         | 13/14254 [00:19<5:38:48,  1.43s/it, loss=1075.60 (897.77), wps=4021, wpb=5703, bsz=131, lr=0.25, clip=100%, gnorm=1689590071670.1538]
| epoch 001:   1%|3                                      | 114/14254 [02:44<5:46:16,  1.47s/it, loss=1859.97 (1038.92), wps=3977, wpb=5695, bsz=169, lr=0.25, clip=100%, gnorm=7599585518747.3330]

[michaelauli]

Those values worked well based on cross validation experiments.

[Zrachel]

Thank you. Is there any explanation on why input should be in a small range (normal(0, 0.1)) instead of normal(0, 1)?