Bert (sentence classification) output is non-deterministic for PyTorch (not for TF)

🐛 Bug

Information

Model I am using (Bert, XLNet …): Bert

Language I am using the model on (English, Chinese …): German

The problem arises when using:

• the official example scripts: (give details below)
• my own modified scripts: (give details below)

The tasks I am working on is:

• an official GLUE/SQUaD task: (give the name)
• my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

1. Load model: config = BertConfig.from_json_file(config_filename) model = BertForSequenceClassification(config) state_dict = torch.load(model_filename) model.load_state_dict(state_dict)
2. Do inference twice on the same input + compare results.
3. Alternatively, save the first output, load the model from scratch, and run the same inference. Even in this case, the first output will not be the same as the next time.

Expected behavior

The prediction value should be deterministic. Note that it is deterministic when the model parameters are loaded from a TensorFlow file (with from_tf=True).

Environment info

• transformers version: 2.10.0
• Platform: Linux-5.3.0-55-generic-x86_64-with-Ubuntu-19.10-eoan
• Python version: 3.7.5
• PyTorch version (GPU?): 1.5.0 (False)
• Tensorflow version (GPU?): 2.0.0 (False)
• Using GPU in script?: no
• Using distributed or parallel set-up in script?: no

[LysandreJik]

Well, it depends. A few things may be responsible here:

• Your model is not in eval mode (model.eval()), resulting in dropout layers affecting your results
• Your fine-tuned model is lacking some layers, which are therefore initialized randomly.

Can you check the logs by putting the following two lines above your model load?

import logging

logging.basicConfig(level=logging.INFO)

Can you also try by using the from_pretrained method (given that your model filename is pytorch_model.bin)?

config = BertConfig.from_json_file(config_filename) 

model = BertForSequenceClassification.from_pretrained(model_dir, config=config)

Or, simpler, if the configuration is in the same folder as your model filename:

model = BertForSequenceClassification.from_pretrained(model_dir)

[LysandreJik]

The logging is useful when you’re loading using from_pretrained as it tells you which layers were not initialized with the model. For example if your checkpoint is a base BERT model that you try to load in the sequence classification model, it will load it but the classifier layer would be randomly initialized. The logging would have told you 😄.

Glad we could resolve your problem!