[LightningLite] Gradient Clipping with AMP

🚀 Feature

Currently, there doesn’t seem to be a way to do gradient clipping when using AMP with LightningLite. This is because the grad scalar code is behind the NativeMixedPrecisionPlugin, and there doesn’t seem to be any way of changing this for LightningLite.

Motivation

Grad clipping is very popular across DL, and having some way to do this is very important.

Pitch

I see that the NativeMixedPrecisionPlugin has functionality for doing this for LightningModules, but not for Lite models.

The method used for LightningModules (using configure_gradient_clipping) – if extended to work with Lite modules – also seems to not be in the spirit of Lite (but it’s better than nothing, I suppose).

---

If you enjoy Lightning, check out our other projects! ⚡

• Metrics: Machine learning metrics for distributed, scalable PyTorch applications.

• Lite: enables pure PyTorch users to scale their existing code on any kind of device while retaining full control over their own loops and optimization logic.

• Flash: The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, fine-tuning, and solving problems with deep learning.

• Bolts: Pretrained SOTA Deep Learning models, callbacks, and more for research and production with PyTorch Lightning and PyTorch.

• Lightning Transformers: Flexible interface for high-performance research using SOTA Transformers leveraging Pytorch Lightning, Transformers, and Hydra.

cc @borda @carmocca @justusschock @awaelchli @akihironitta @rohitgr7

[carmocca]

I wouldn’t introduce any hooks to Lite, but we could expose a clip_gradients function that always works the selected Lite configuration.

[awaelchli]

I think providing a Lite.clip_gradients() could work. This could even be useful for manual optimization from within the LightningModule.