Add more options to create_supervised_trainer

🚀 Feature

Idea is to add more options to create_supervised_trainer helper:

• native AMP option similar to https://github.com/lidq92/LinearityIQA/blob/master/modified_ignite_engine.py
  • we may either use nvidia/apex if installed or torch native amp if v1.6.0. Otherwise, raise an error.
• grads cliping as proposed #419

For Hacktoberfest/PyDataGlobal contributors, feel free to ask questions for details if any and say that you would like to tackle the issue. Please, take a look at CONTRIBUTING guide.

[vfdev-5]

Cross-posted from #1589

---

We discussed this PR and related issue with the team and we think that we should explore a bit different approach. Helper method create_supervised_trainer is roughly made of 2 things : update function definition and Engine setup.

Probably, it would more helpful to provide public methods like:

• supervised_training_step
• supervised_training_step_tpu
• supervised_training_step_apex
• supervised_training_step_amp

and inside create_supervised_trainer we could setup the trainer according to provided options without lots of if/else. Maybe, we can skip for instance grad norm.

Basically, the idea is something like that :

def get_training_step_1(a):
    def training_step(e, b):
        print(a, e, b)
    return training_step
    
def get_training_step_2(a):
    def training_step(e, b):
        print(a, e, b, "with amp")
    return training_step

def create_supervised_trainer(a, opt):
    training_step = None
    if opt == 1:
        training_step = get_training_step_1(a)
    elif opt == 2:
        training_step = get_training_step_2(a)
        
    e = Engine(training_step)
    return e  

[apthagowda97]

Sorry I was busy from past few days. I ll try to finish it Asap.