run_lm_finetuning.py does not define a do_lower_case argument

The file references args.do_lower_case, but doesn’t have the corresponding parser.add_argument call.

As an aside, has anyone successfully applied LM fine-tuning for a downstream task (using this code, or maybe using the original tensorflow implementation)? I’m not even sure if the code will run in its current state. And after fixing this issue locally, I’ve had no luck using the output from fine-tuning: I have a model that gets state-of-the-art results when using pre-trained BERT, but after fine-tuning it performs no better than omitting BERT/pre-training entirely! I don’t know whether to suspect that there are might be other bugs in the example code, or if the hyperparameters in the README are just a very poor starting point for what I’m doing.

[nikitakit]

I figured out why I was seeing such poor results while attempting to fine-tune: the example saves model.bert instead of model to pytorch_model.bin, so the resulting file can’t just be zipped up and loaded with from_pretrained.

[nikitakit]

Thanks for fixing these!

After addressing the save/load mismatch I’m seeing downstream performance comparable to using pre-trained BERT. I just got a big scare when the default log level wasn’t high enough to notify me that weights were being randomly re-initialized instead of loaded from the file I specified. It’s still too early for me to tell if there are actual benefits to fine-tuning, though.