Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Native automatic mixed precision for Catalyst

See original GitHub issue

Native automatic mixed precision support (torch.cuda.amp) is merged:

Not sure if you used Apex Amp, but I know it has many pain points (extension builds, forward/backward compatibilty, DataParallel support, flaky checkpointing, i don’t even know if it can be hacked to handle double backward/gradient penalty, others…). torch.cuda.amp fixes all these, the interface is more flexible and intuitive, and the tighter integration with pytorch brings more future optimizations into scope.

I think the torch.cuda.amp API is a good fit for a higher-level library because its style is more functional (as in, it doesn’t statefully alter anything outside itself). The necessary torch.cuda.amp calls don’t have silent/weird effects elsewhere.

If you want to talk about adding torch.cuda.amp to Catalyst, with an eye towards it becoming the future-proof source of mixed precision, message me on Pytorch slack anytime. I also pinged you there but I’m not sure if you monitor it habitually.

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Reactions:2
  • Comments:8 (4 by maintainers)

github_iconTop GitHub Comments

mcarillicommented, Apr 6, 2020

It’s in master and nightlies. Topk with fp16 tensors is broken on master right now though (unrelated to amp), so I’d give it a week or two before trying. 1.6 will be the first stable release that contains the full torch.cuda.amp.

(Unfortunately I missed the WIP 1.5 release branch due to a tragic comedy: autocast’s initial merge was before the 1.5 fork, but got reverted due to a buggy visual studio 2017 build that wasn’t visible in external CI. I wrote a fix that made the buggy VS build happy, but by then it was too late: 1.5 had already forked, and we couldn’t remerge because the rule is no big feature merges after fork.)

Scitatorcommented, Apr 4, 2020

Thanks for reaching out! We will definitely add this feature. Btw, when should we expect PyTorch release with native automatic mixed precision support ?

Read more comments on GitHub >

github_iconTop Results From Across the Web

Mixed precision training — Catalyst 21.11rc0 documentation
Catalyst support a variety of backends for mixed precision training. ... If you would like to use native AMP support, you could do...
Read more >
Introducing native PyTorch automatic mixed precision for ...
An open source machine learning framework that accelerates the path from research prototyping to production deployment.
Read more >
Automatic Mixed Precision for Deep Learning
With Automatic Mixed Precision, we've realized a 50% speedup in TensorFlow-based ASR model training without loss of accuracy via a minimal code change....
Read more >
[D] Automatic mixed precision training is coming in PyTorch 1.6
For the unfamiliar, mixed precision training is the technique of using lower-precision types (e.g. in PyTorch, using fp16 instead of the default ...
Read more >
Catalyst — A PyTorch Framework for Accelerated Deep ...
It defines the logic of hardware communication and different deep learning techniques usage like distributed or mixed-precision training.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found