How to use l2l.algorithms.MAML correctly with nn.DistributedDataParallel?

This work is awesome!

Using nn.DistributedDataParallel in the following way will raise Error when execute learner = maml.clone() How to use it correctly? Should I use nn.DistributedDataParallel on MyModel and then use MAML? Thanks!

model = MyModel() maml = l2l.algorithms.MAML(model, lr=0.5) model = nn.DistributedDataPrallel(model, device_ids=[rank]) … learner = maml.clone()

[seba-1511]

Hello @AyanamiReiFan, and thanks for the kind words.

Parallelizing MAML with DistributedDataParallel is a bit tricky as the implementation relies on gradient hooks which don’t play well with clone/grad. If you want to use torch.distributed to parallelize the training loop, cherry’s Distributed optimizer is another option:

opt = optim.Adam(model.parameters())
opt = Distributed(model.parameters(), opt, sync=1)
# Training code
opt.step()

If you want to parallelize the model over GPUs, I would use torch.nn.DataParallel:

learner = maml.clone()
learner = torch.nn.DataParallel(learner, device_ids=[0, 1])
# Training code

Let me know if you ever find a solution to using DistributedDataParallel, I’d be curious to know the solution.

[janbolle]

@Kulbear

1. I did not use a GPU version as the batch-sizes and NNs are relatively small and I suppose this won’t speed things up much in this setting - but you could easily implement it since Ray also supports GPU
2. I only did experiments for Regression and Classification. Also, the implementation is done using TF2.0. Would the implementation be helpful for you?