Setting different lr for different parameters using fairseq command-line

❓ Questions and Help

Before asking:

1. search the issues.
2. search the docs.

What is your question?

Is it possible to use the fairseq-train command-line tools to set different learning rates for different parameters?

Code

What have you tried?

I have searched the issue and docs but cannot find useful suggestions. Thanks for your reply!!!

What’s your environment?

• fairseq Version (e.g., 1.0 or main):
• PyTorch Version (e.g., 1.0)
• OS (e.g., Linux):
• How you installed fairseq (pip, source):
• Build command you used (if compiling from source):
• Python version:
• CUDA/cuDNN version:
• GPU models and configuration:
• Any other relevant information:

[ZeroYuHuang]

No you cannot. I doubt pytorch support this feature.

I believe having more hyper parameters (lr for each weight) is not so welcomed. That does mean few people do this. So wish you good luck.

Sorry I was wrong. Pytorch can specify learning rate for each paramter.

And yes, you can. Every class with a @register_ decoration can be invoked by proper fairseq’s command line tool. So through fairseq-train you can invoke your custom optimizer.

That helps a lot thanks. I have looked up the code in fairseq, it supports to customize our own optimizer through @register_optimizer . Thanks for reply!!! And I will close this issue since problems solved.

[gmryu]

@ZeroYuHuang I am sorry. I was wrong. There is an optimizer called composite

You can use this to set different lr for different parameter groups.