'AmpOptimizerState' object has no attribute 'all_fp32_params'

I initialized the model and optimizer with amp

optimizer = Adam(filter(lambda p: p.requires_grad, model.parameters()), args.lr)
scheduler = ReduceLROnPlateau(optimizer, patience=1, factor=0.1, verbose=True, mode='max')
model, optimizer = amp.initialize(model, optimizer, opt_level="O1", verbosity=0)

and then I tried to add a group of params to the optimizer before training

optimizer.add_param_group({'params':unfreezed_params, 'lr':lr})

I got the following error

AttributeError: 'AmpOptimizerState' object has no attribute 'all_fp32_params'

System info: Ubuntu 18.04, cuda 10.0.130, pytorch 1.1.0

[adriansahlman]

@thuwyh Quick workaround is to do the following:

optimizer = Adam(filter(lambda p: p.requires_grad, model.parameters()), args.lr)
scheduler = ReduceLROnPlateau(optimizer, patience=1, factor=0.1, verbose=True, mode='max')
model, optimizer = amp.initialize(model, optimizer, opt_level="O1", verbosity=0)
optimizer._lazy_init_maybe_master_weights()

[mcarilli]

I think I know what’s going on. Amp relies on some internal stash that has knowledge of the parameters. Typically this is silently lazy-initialized by the backward context manager, or an initial call to optimizer.zero_grad().

I anticipated that people would only want to add param groups later on in training, after many steps had already been taken and the lazy stash initialization had taken place, so that is what the tests cover. I will make sure add_param_group also implements initialization if necessary.