Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

LRFinder w/ Gradient Accumulation

See original GitHub issue

Great package! Thank you for sharing 😃

I was wondering if you plan on adding gradient accumulation support for using LRFinder with a larger batch size.
Will you be adding mixed precision support?

Issue Analytics

State:
Created 4 years ago
Comments:11 (9 by maintainers)

Top GitHub Comments

2reactions

davidtvscommented, Dec 23, 2019

The PR from @NaleRaphael is merged. Thanks @rsomani95 for raising the issue.

1reaction

NaleRaphaelcommented, Dec 20, 2019

Hi @rsomani95 . Many thanks for your help and feedback, and I’m glad that these implementation helped!

And it’s quite weird that it takes longer time to run when torch.backends.cudnn.benchmark = True. As far as I know, that flag should accelerate training speed when input size is fixed in each iteration.

However, it seems to me that it’s not harmful to pend the issue about torch.backends.cudnn.benchmark currently. Because it’s not related to LRFinder directly and the use of it depends on user. Though, I’ll keep it in mind!

Besides, it seems that apex is going to be integrated as a builtin component of PyTorch in the future. (nvidai/apex#659) I will keep tracking this, too.

@davidtvs Before merging this PR, I would like to add some code to make users able to install apex optionally. I’ll leave a comment here when it’s done.

Thanks you, guys!

Top Results From Across the Web

Effective Training Techniques - PyTorch Lightning

Accumulated gradients run K small batches of size N before doing a backward pass. The effect is a large effective batch size of...

LR finder and small batch sizes - fast.ai Course Forums

One possible solution might be to utilize gradient accumulation to simulate higher batch sizes just for calculating the learning rate. However, ...

Source code for monai.optimizers.lr_finder

... gradients are not accumulated. non_blocking_transfer: when `True`, moves data to device asynchronously if possible, e.g., moving CPU Tensors with pinned ...

Performing gradient accumulation with Accelerate

This is done by accumulating gradients over several batches, and only stepping the optimizer after a certain number of batches have been performed....

LR Finder Using PyTorch - Kaggle

This Python 3 environment comes with many helpful analytics libraries ... also: # https://nvidia.github.io/apex/advanced.html#gradient-accumulation-across- ...