Finetune on multi-GPUs

I want to finetune the models on multi-GPUs, as following the office documents, I try this: model = make_model( args.model_name, pretrained=True, num_classes=len(classes), dropout_p=args.dropout_p, use_original_classifier=True ) model = nn.DataParallel(model) model = model.to(device)

But get error as: Traceback (most recent call last): File "pg_cls.py", line 83, in <module> mean=model.original_model_info.mean, File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __getattr__ type(self).__name__, name)) AttributeError: 'DataParallel' object has no attribute 'original_model_info'

[creafz]

For now, you can use standard PyTorch methods to freeze some layers:

Create a new model:

model = make_model('resnet18', num_classes=10, pretrained=True)

To freeze all feature extracting layers and finetune only the classifer:

for param in model._features.parameters():
    param.requires_grad = False

To freeze only some layers of the feature extractor:

# The number of feature extracting layers that should be finetuned. 
# Here we want to finetune the 2 last layers of the feature extractor and the classifying layers
NUM_LAYERS_TO_UNFREEZE = 2

# Just to be sure, print those layers that will be frozen...
print(list(model._features.children())[:-NUM_LAYERS_TO_UNFREEZE])

# ... and those layers that we want to finetune
print(list(model._features.children())[-NUM_LAYERS_TO_UNFREEZE:])

# Finally freeze layers
for module in list(model._features.children())[:-NUM_LAYERS_TO_UNFREEZE]:
    for param in module.parameters():
        param.requires_grad = False

# Print parameters that will be updated during training.
print([name for (name, param) in model.named_parameters() if param.requires_grad])

You will also need to pass only parameters from ‘unfrozen’ layers to an optimizer or you will get an exception

optimizer = optim.SGD((param for param in model.parameters() if param.requires_grad), lr=..., momentum=...)

I will think about how this functionality could be added to the library.

[Ken2yLiu]

@creafz sorry for replying late, your answers help a lot, then I would close this issue now