Bert for passage reranking

Hi I am currently trying to implement bert for passage reranking in pytorch. Here is the paper and github repo. https://arxiv.org/abs/1901.04085 https://github.com/nyu-dl/dl4marco-bert

I’ve downloaded their bert large model checkpoint and bert config for the task the convert_tf_checkpoint_to_pytorch function seems to successfully extract the weights from tensorflow.

Then while initialising the pytorch model

Initialize PyTorch weight ['bert', 'pooler', 'dense', 'kernel']
Skipping bert/pooler/dense/kernel/adam_m
Skipping bert/pooler/dense/kernel/adam_v
Skipping global_step

     35 
     36     # Load weights from tf checkpoint
---> 37     load_tf_weights_in_bert(model, tf_checkpoint_path)
     38 
     39     # Save pytorch-model

~/anaconda3/envs/new_fast_ai/lib/python3.7/site-packages/pytorch_pretrained_bert/modeling.py in load_tf_weights_in_bert(model, tf_checkpoint_path)
     88                 pointer = getattr(pointer, 'weight')
     89             elif l[0] == 'output_bias' or l[0] == 'beta':
---> 90                 pointer = getattr(pointer, 'bias')
     91             elif l[0] == 'output_weights':
     92                 pointer = getattr(pointer, 'weight')

~/anaconda3/envs/new_fast_ai/lib/python3.7/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
    533                 return modules[name]
    534         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 535             type(self).__name__, name))
    536 
    537     def __setattr__(self, name, value):

AttributeError: 'BertForPreTraining' object has no attribute 'bias'

I assume it is issues with the final layer What is the best way for me to go about resolving this?

thanks in advance!

[pertschuk]

Update for latest transformers, add modeling_bert.py:78:

    for name, array in zip(names, arrays):
        if name in ['output_weights', 'output_bias']:
            name = 'classifier/' + name

and convert_bert_original_tf_checkpoint_to_pytorch.py

config.num_labels = 2
    print("Building PyTorch model from configuration: {}".format(str(config)))
    model = BertForSequenceClassification(config)

[thomwolf]

The convert_tf_checkpoint_to_pytorch script is made to convert the Google pre-trained weights in BertForPretraining model, you have to modify it to convert another type model.

In your case, you want to load the passage re-ranking model in a BertForSequenceClassification model which has the same structure (BERT + a classifier on top of the pooled output) as the NYU model.

here is a quick way to do that:

• install pytorch-pretrained-bert from source so you can modify it
• change https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/convert_tf_checkpoint_to_pytorch.py#L34 to initialize a BertForSequenceClassification model instead of the BertForPreTraining model in the conversion script.
• the structure is not exactly identical so you need to ADD a line that say pointer = getattr(pointer, 'cls') in the TWO if-conditions related to output_weights and output_bias (between L89 and L90 and between L91 and L92 in modeling.py here: https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/modeling.py#L90 and https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/modeling.py#L92).
• this should let you convert the tensorflow model in a pytorch one using the scripts.