Native automatic mixed precision for Catalyst

Native automatic mixed precision support (torch.cuda.amp) is merged: https://pytorch.org/docs/master/amp.html https://pytorch.org/docs/master/notes/amp_examples.html

Not sure if you used Apex Amp, but I know it has many pain points (extension builds, forward/backward compatibilty, DataParallel support, flaky checkpointing, i don’t even know if it can be hacked to handle double backward/gradient penalty, others…). torch.cuda.amp fixes all these, the interface is more flexible and intuitive, and the tighter integration with pytorch brings more future optimizations into scope.

I think the torch.cuda.amp API is a good fit for a higher-level library because its style is more functional (as in, it doesn’t statefully alter anything outside itself). The necessary torch.cuda.amp calls don’t have silent/weird effects elsewhere.

If you want to talk about adding torch.cuda.amp to Catalyst, with an eye towards it becoming the future-proof source of mixed precision, message me on Pytorch slack anytime. I also pinged you there but I’m not sure if you monitor it habitually.

[mcarilli]

It’s in master and nightlies. Topk with fp16 tensors is broken on master right now though (unrelated to amp), so I’d give it a week or two before trying. 1.6 will be the first stable release that contains the full torch.cuda.amp.

(Unfortunately I missed the WIP 1.5 release branch due to a tragic comedy: autocast’s initial merge was before the 1.5 fork, but got reverted due to a buggy visual studio 2017 build that wasn’t visible in external CI. I wrote a fix that made the buggy VS build happy, but by then it was too late: 1.5 had already forked, and we couldn’t remerge because the rule is no big feature merges after fork.)

[Scitator]

Thanks for reaching out! We will definitely add this feature. Btw, when should we expect PyTorch release with native automatic mixed precision support ?