Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Seq2seq finetune example: "Please save or load state of the optimizer"

See original GitHub issue

When running the example scripts in examples/seq2seq/finetune_bart and finetune_t5, get warning messages:

Environment info

transformers version: 3.3.1
Platform: Linux-4.15.0-66-generic-x86_64-with-glibc2.10
Python version: 3.8.5
PyTorch version (GPU?): 1.6.0 (True)
Tensorflow version (GPU?): not installed (NA)
Using GPU in script?: Ran both with and without gpus; same result
Using distributed or parallel set-up in script?: no

Who can help

@sshleifer for examples/seq2seq, Bart @patrickvonplaten (maybe because this also happens in T5?)

Information

Model I am using (Bert, XLNet …): Occurs when running bart and also when running T5 via the examples/seq2seq/finetune

The problem arises when using:

the official example scripts: (give details below)
my own modified scripts: (give details below)

The tasks I am working on is:

[X ] an official GLUE/SQUaD task: (give the name)
my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior: Steps to reproduce:

clone transformers into new directory
Set up environment (new): cd transformers && pip install .e; cd examples && pip install -r requirements.txt
cd seq2seq && ./finetune_t5_bart_tiny.sh

Observe that warnings are printed:

…/python3.8/site-packages/pytorch_lightning/utilities/distributed.py:37: UserWarning: Could not log computational graph since the model.example_input_array attribute is not set or input_array was not given warnings.warn(*args, **kwargs) …/python3.8/site-packages/torch/optim/lr_scheduler.py:200: UserWarning: Please also save or load the state of the optimzer when saving or loading the scheduler. warnings.warn(SAVE_STATE_WARNING, UserWarning)

(There is both the optimizer warning and the computational graph logging warning)

Expected behavior

Should not see warnings for the given example.

Other notes:

There was a related issue where supplementary files / checkpoints were not being saved, but that seems to be fixed now.

Issue Analytics

State:
Created 3 years ago
Reactions:1
Comments:5 (5 by maintainers)

Top GitHub Comments

1reaction

LysandreJikcommented, Oct 15, 2020

I believe this warning has been hidden on the master branch, and will be hidden in the next release. See this.

0reactions

jsroznercommented, Oct 15, 2020

Cool! Looks like originally in #7401. I pulled master and confirmed that this is fixed.

Any notes on the computational graph warning that also pops up? …/python3.8/site-packages/pytorch_lightning/utilities/distributed.py:37: UserWarning: Could not log computational graph since the model.example_input_array attribute is not set or input_array was not given

Top Results From Across the Web

Fine-tune a pretrained model - Hugging Face

In this tutorial, you will fine-tune a pretrained model with a deep learning ... from tensorflow.keras.optimizers import Adam # Load and compile our...

Saving and loading multiple models in one file using PyTorch

Steps · 1. Import necessary libraries for loading our data · 2. Define and intialize the neural network · 3. Initialize the optimizer...

Save and load model optimizer state - python - Stack Overflow

It is a bit too complicated of an example to share, but in short, I am not able to use model.save('model_file.h5') and keras.models.load_model(' ......

Checkpointing Distributed Models and Optimizer States

The following code shows an example of how to use the checkpoint APIs for saving and loading a model trained with model parallelism....

Neural machine translation with attention | Text - TensorFlow

This tutorial demonstrates how to train a sequence-to-sequence ... If you would like to use Nvidia GPU with TensorRT, please make sure the ......