Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

add gradient normalization and accumulation in supervised_training_step_* functions

See original GitHub issue

🚀 Feature

After #1589 , grad_norm is left to add (#419), and also gradient accumulation is also common in supervised training, we should also add that to have full supervised training.

Issue Analytics

State:
Created 3 years ago
Reactions:2
Comments:5 (3 by maintainers)

Top GitHub Comments

1reaction

vfdev-5commented, Sep 25, 2021

@Ishan-Kumar2 I think a flag to existing supervised_training_step* can be a good option.

1reaction

ydcjeffcommented, Feb 22, 2021

Hi @KickItLikeShika Sure, feel free to go ahead

Top Results From Across the Web

[1710.02368] Accumulated Gradient Normalization - arXiv

This work addresses the instability in asynchronous data parallel optimization. It does so by introducing a novel distributed optimizer which is ...

A.3 Normalized Gradient Descent

Normalizing out the full gradient magnitude we can interpret our fully normalized step as a standard gradient descent step with a steplength /...

Accumulated Gradient Normalization

Abstract. This work addresses the instability in asynchronous data parallel optimization. It does so by introducing a novel distributed optimizer which is ...

Lecture 7 | Training Neural Networks II - YouTube

Gradient descent, how neural networks learn | Chapter 2, Deep learning. 3Blue1Brown ... Lecture 3 | Loss Functions and Optimization.

Gradient Accumulation Normalisation : r/pytorch - Reddit

optimizer = ... NUM_ACCUMULATION_STEPS = ... for epoch in range(...): for idx, sample in enumerate(dataloader): inputs, labels = sample…