Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Bert Checkpoint Breaks 3.02 -> 3.1.0 due to new buffer in BertEmbeddings

See original GitHub issue

Hi,

Thanks for the great library. I noticed this line being added (https://github.com/huggingface/transformers/blob/v3.1.0/src/transformers/modeling_bert.py#L190) in the latest update.

It breaks checkpoints that were saved when this line wasn’t there.

	Missing key(s) in state_dict: "generator_model.electra.embeddings.position_ids", "discriminator_model.electra.embeddings.position_ids".

Issue Analytics

State:
Created 3 years ago
Reactions:4
Comments:28 (10 by maintainers)

Top GitHub Comments

23reactions

LysandreJikcommented, Sep 2, 2020

You can also use the load_state_dict method with the strict option set to False:

model.load_state_dict(state_dict, strict=False)

4reactions

dfdazaccommented, Jul 22, 2021

I think it’s safe to use model.load_state_dict(state_dict, strict=False) if the only missing information is the position_ids buffer. This tensor is indeed used by the model, but it’s just a constant tensor containing a list of integers from 0 to the maximum number of position embeddings. The tensor is first created in the constructor of the BertEmbeddings class, in this line:

https://github.com/huggingface/transformers/blob/fcf83011dffce3f2e8aad906f07c1ec14668f877/src/transformers/models/bert/modeling_bert.py#L182

As such, it’s not really part of the optimizable parameters of the model. This means that it doesn’t matter if position_ids is not available when calling load_state_dict, because the line above will create it anyway in the constructor with the required values.