Hyperparameters of AdamW

Table 2 of your paper shows that AdamW on ImageNet is as good as SGDM, which is very excited. Would like to share with us the hyperparameters? Thx!

• I guess from your paper ResNet + AdamW is AdamW(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0.0001, amsgrad=False), Is it right? However, I have done an experiment with the above setting and it was two points lower than your result. I’m confused

• What is the hyperparameter of MobileNetV2+AdamW

[bhheo]

Hi

Thank you for your interest in our paper.

For torch.optim.AdamW, you have to use weight_decay=0.1. In AdamW paper, they decoupled the weight decay which means w = (1 - weight_decay)w But, PyTorch implementation is w = (1 - lr * weight_decay) w https://github.com/pytorch/pytorch/blob/b31f58de6fa8bbda5353b3c77d9be4914399724d/torch/optim/adamw.py#L73 It makes it easy to utilize the learning rate scheduler for weight decay but requires changing parameters.

In the paper, we followed the notation of AdamW paper. So lr=1e-3, weight_decay=0.1 is the PyTorch parameter for weight decay 1e-4.

You can find a similar setting on NovoGrad paper https://arxiv.org/pdf/1905.11286.pdf

[bhheo]

5e-3 is correct. torch.optim.AdamW(param, lr=2e-3, weight_decay=5e-3)

It is 1e-5 in paper notation.