Where I can find official pretrained weights of SOP in Albert and NSP in Bert?

Hi Guys,

I was checking for the pre-trained weights ( 2 layer classifier ) of SOP in Albert and NSP in Bert. Seems like it is initializing randomly every time.

Can we have the official weights loaded here or is it not available from official models? Can anyone clarify please.

Thanks.

[LysandreJik]

Hi @s4sarath, sorry for the delayed response. If the checkpoints on the hub do not satisfy you (I see the SOP/NSP layers are indeed lacking), conversion scripts are available for each model:

• BERT, see the convert_* scripts
• ALBERT

I confirm this successfully exports the full model, including the NSP/SOP weights:

ALBERT

wget https://storage.googleapis.com/albert_models/albert_base_v2.tar.gz 
tar -xzf albert_base_v2.tar.gz 
python convert_albert_original_tf_checkpoint_to_pytorch.py --tf_checkpoint_path=albert_base/model.ckpt-best --albert_config_file=albert_base/albert_config.json --pytorch_dump_path=albert_base/pytorch_model.bin
cp albert_base/albert_config.json albert_base/config.json   

>>> from transformers import TFAlbertForPreTraining
>>> model = TFAlbertForPreTraining.from_pretrained("albert_base", from_pt=True)
[...]
All the weights of TFAlbertForPreTraining were initialized from the PyTorch model.
If your task is similar to the task the model of the checkpoint was trained on, you can already use TFAlbertForPreTraining for predictions without further training.

BERT

Same for BERT:

wget https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip
unzip uncased_L-12_H-768_A-12.zip  
python convert_bert_original_tf_checkpoint_to_pytorch.py --tf_checkpoint_path=uncased_L-12_H-768_A-12/bert_model.ckpt --bert_config_file=uncased_L-12_H-768_A-12/bert_config.json --pytorch_dump_path=uncased_L-12_H-768_A-12/pytorch_model.bin

>>> from transformers import TFBertForPreTraining
>>> bert = TFBertForPreTraining.from_pretrained("uncased_L-12_H-768_A-12", from_pt=True)
[...[
All the weights of TFBertForPreTraining were initialized from the PyTorch model.
If your task is similar to the task the model of the checkpoint was trained on, you can already use TFBertForPreTraining for predictions without further training.

Hope that helps.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.