New run_language_modeling.py continuing trainng

I tried using new script for Language Modeling to train model from scratch.

I was training model with old script. When I tried continuing using new one, there is no option for --should_continue.

Does --overwrite_output_dir makes it train from scratch if i use same directory for --model_name_or_path and --output_dir?

So, I am continuing training with old script.

[julien-c]

Yes you would use --overwrite_output_dir (it was implicitly added before)

You would do:

--model_name_or_path ./model_name/checkpoint-9000
--output_dir ./model_name
--overwrite_output_dir

If this is a frequent request we could add it back but I feel like it was more confusing than anything.

[marscod]

I am using TrainingArguments but couldn’t continue training from last checkpoint. It is starting from step 0. Any suggestion?

training_args = TrainingArguments(
    output_dir=last_checkpoint,
    overwrite_output_dir=True,
...
)
 trainer = Trainer( model=model, #roberta config model
           args=training_args,
...
)

e.g., last_checkpoint="/saved_model/checkpoint-20000/" @julien-c