Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Add `weight_decay_filter` and `lars_adaptation_filter` to LARS
See original GitHub issue🚀 Feature
Add weight_decay_filter and lars_adaptation_filter to LARS
Motivation
weight decay typically shouldn’t be applied to BatchNorm. See fast.ai and this pytorch discuss thread.
The facebook vicreg code has parameters weight_decay_filter and lars_adaptation_filter which they set to True for any parameter that has ndim 1.
Pitch
There should be a simple way to disable weight decay and LARS adaptation on ndim==1 parameters.
Alternatives
Port Facebook LARS code and use it instead of lightning flash LARS code.
Issue Analytics
- State:
- Created a year ago
- Reactions:1
- Comments:8 (5 by maintainers)
Top Results From Across the Web
Top Related Medium Post
Top Related StackOverflow Question
Troubleshoot Live Code
Top Related Reddit Thread
Top Related Hackernoon Post
Top Related Tweet
Top Related Dev.to Post
Top Related Hashnode Post
@krshrimali Thanks! And I am happy to help with code review if you tag me in the PR
@krshrimali Great! I am following this issue.