PiecewiseLinear dict support

In ignite 0.2.1, PiecewiseLinear didn’t expect the optimizer to be strictly a sub-class of torch.optim.Optimizer and allowed generic parameter group dicts as well.

But it looks like in later versions, this generic support was withdrawn.

What’s the context behind this change?

I had been successfully using PiecewiseLinear from ignite 0.2.1 along with Microsoft’s DeepSpeed ZeRO optimizer, which isn’t strictly a sub-class of torch.optim.Optimizer and Microsoft addressed the reasons for this in their repo. However, due to some useful features in later versions of ignite, we decided to upgrade to a newer version of ignite (0.3 and later also 0.4). In the process of this upgrade, we realized PiecewiseLinear no longer works with the ZeRO optimizer.

It would be great if support for generic parameter group dicts is added back.

[g-karthik]

@vfdev-5 yes, I am using model_engine during training and model for evaluation, that’s correct. As for DDP, you can use the dist_init_required flag in initialize() for it.

[g-karthik]

@vfdev-5 Yes, deepspeed is just using torch.distributed with NCCL backend under the hood. I haven’t tried it for smaller models but there will be reduced memory footprint even with those, so you could train those smaller models with larger batch sizes and longer sequence lengths.