AttributeError: 'BertForPreTraining' object has no attribute 'shape'

Is there any suggestion for fixing the following? I was trying “convert_tf_checkpoint_to_pytorch.py” to convert a model trained from scratch but the conversion didn’t work out…

Skipping cls/seq_relationship/output_weights/adam_v
Traceback (most recent call last):
  File "pytorch_pretrained_bert/convert_tf_checkpoint_to_pytorch.py", line 66, in <module>
    args.pytorch_dump_path)
  File "pytorch_pretrained_bert/convert_tf_checkpoint_to_pytorch.py", line 37, in convert_tf_checkpoint_to_pytorch
    load_tf_weights_in_bert(model, tf_checkpoint_path)
  File "/content/my_pytorch-pretrained-BERT/pytorch_pretrained_bert/modeling.py", line 117, in load_tf_weights_in_bert
    assert pointer.shape == array.shape
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 535, in __getattr__
    type(self).__name__, name))
AttributeError: 'BertForPreTraining' object has no attribute 'shape'

[jbrry]

I’m getting a similar error when trying to convert the newer BERT models released at tensorflow/models/tree/master/official/nlp/.

These models are either BERT models trained with Keras or else checkpoints converted from the original google-research/bert repository. I also get the same error when I convert the TF1 to TF2 checkpoints myself using the tf2_encoder_checkpoint_converter.py script:

What I have tried:

First, I have downloaded a model:

wget https://storage.googleapis.com/cloud-tpu-checkpoints/bert/keras_bert/cased_L-12_H-768_A-12.tar.gz
# or
wget https://storage.googleapis.com/cloud-tpu-checkpoints/bert/tf_20/cased_L-12_H-768_A-12.tar.gz 

After unpacking:

export BERT_BASE_DIR=cased_L-12_H-768_A-12

transformers-cli convert --model_type bert \
    --tf_checkpoint $BERT_BASE_DIR/bert_model.ckpt \
    --config $BERT_BASE_DIR/bert_config.json \
    --pytorch_dump_output $BERT_BASE_DIR/pytorch_model.bin

The command prints the configuration but throws the following error:

INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_attention_layer/_value_dense/kernel/.ATTRIBUTES/VARIABLE_VALUE with shape [768, 12, 64]
INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_attention_layer_norm/beta/.ATTRIBUTES/VARIABLE_VALUE with shape [768]
INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_attention_layer_norm/gamma/.ATTRIBUTES/VARIABLE_VALUE with shape [768]
INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_attention_output_dense/bias/.ATTRIBUTES/VARIABLE_VALUE with shape [768]
INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_attention_output_dense/kernel/.ATTRIBUTES/VARIABLE_VALUE with shape [12, 64, 768]
INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_intermediate_dense/bias/.ATTRIBUTES/VARIABLE_VALUE with shape [3072]
INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_intermediate_dense/kernel/.ATTRIBUTES/VARIABLE_VALUE with shape [768, 3072]
INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_output_dense/bias/.ATTRIBUTES/VARIABLE_VALUE with shape [768]
INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_output_dense/kernel/.ATTRIBUTES/VARIABLE_VALUE with shape [3072, 768]
INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_output_layer_norm/beta/.ATTRIBUTES/VARIABLE_VALUE with shape [768]
INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_output_layer_norm/gamma/.ATTRIBUTES/VARIABLE_VALUE with shape [768]
INFO:transformers.modeling_bert:Loading TF weight save_counter/.ATTRIBUTES/VARIABLE_VALUE with shape []
INFO:transformers.modeling_bert:Skipping _CHECKPOINTABLE_OBJECT_GRAPH
Traceback (most recent call last):
  File "/home/jbarry/anaconda3/envs/transformers/bin/transformers-cli", line 30, in <module>
    service.run()
  File "/home/jbarry/anaconda3/envs/transformers/lib/python3.6/site-packages/transformers/commands/convert.py", line 62, in run
    convert_tf_checkpoint_to_pytorch(self._tf_checkpoint, self._config, self._pytorch_dump_output)
  File "/home/jbarry/anaconda3/envs/transformers/lib/python3.6/site-packages/transformers/convert_bert_original_tf_checkpoint_to_pytorch.py", line 36, in convert_tf_checkpoint_to_pytorch
    load_tf_weights_in_bert(model, config, tf_checkpoint_path)
  File "/home/jbarry/anaconda3/envs/transformers/lib/python3.6/site-packages/transformers/modeling_bert.py", line 118, in load_tf_weights_in_bert
    assert pointer.shape == array.shape
  File "/home/jbarry/anaconda3/envs/transformers/lib/python3.6/site-packages/torch/nn/modules/module.py", line 585, in __getattr__
    type(self).__name__, name))
AttributeError: 'BertForPreTraining' object has no attribute 'shape'

This is happening in a fresh environment with PyTorch 1.3 installed in Anaconda (Linux), as well as pip-installing tf-nightly and transformers (2.3.0).

Has anyone else been able to successfully convert the TF 2.0 version models to PyTorch or know where I’m going wrong? Thanks!

[lironT74]

@thomwolf If the above fix will be added to the master branch this will be great https://github.com/smartshark/transformers/pull/1