Do we need to set mixed-precision explicitly or is it handled if tensor cores available?

I following your awesome guide on timm: https://towardsdatascience.com/getting-started-with-pytorch-image-models-timm-a-practitioners-guide-4e77b4bf9055.

I am running training on an A100-based VM which should support mixed-precision training. Does Trainer from PyTorch Accelerated take care of that automatically?

[sayakpaul]

Thanks so much for being thorough with your explanations. And yes it makes sense to use config files to launch training whenever possible.

[Chris-hughes10]

No problem. If you would like to enable mixed precision while launching with python main.py you can do so in two ways:

1. The easiest is to set the Environment Variable USE_FP16 as ‘true’ (or ‘True’, it doesn’t matter). This is what the launcher does for you based on the value in your config file.
2. Hard code the value in the Trainer so that it always uses mixed precision (this is not really recommended though)

If you want to go ahead with option 2, you will need to update the Trainer’s accelerator object, which is what handles moving data between devices. You can override the create_accelerator method to do this. Here is an example of how this would look:

from accelerate import Accelerator
from pytorch_accelerated import Trainer

class Fp16Trainer(Trainer):

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

    def _create_accelerator(self):
        return Accelerator(fp16=True)

That seems a lot of effort for such a small change though, so it wasn’t really the intended approach for this!