Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

LM fine tuning on top of a custom model

See original GitHub issue

Currently the finetune_on_pregenerated.py script allows only fine-tuning on top of one of the five pretrained bert models. However I don’t understand why there is such a restriction. I am trying to finetune a LM on top of a custom bert model (mt-dnn). Of course I can just remove the choices, but I am wondering if there is some rationale behind this, as none of the other example scripts contains this restriction.

https://github.com/huggingface/pytorch-pretrained-BERT/blob/94980b529fad34f55c5d34c94bae2814db6773a6/examples/lm_finetuning/finetune_on_pregenerated.py#L126-L128

Issue Analytics

State:
Created 4 years ago
Comments:7 (6 by maintainers)

Top GitHub Comments

1reaction

sivakumarchcommented, Nov 10, 2019

is this change applicable and working with run_lm_finetuning.py in 2.x?

0reactions

thomwolfcommented, Apr 11, 2019

Indeed, I’ll take care of it. I’m on the repo now anyway.

Top Results From Across the Web

Fine-tune a pretrained model - Hugging Face

This is known as fine-tuning, an incredibly powerful training technique. In this tutorial, you will fine-tune a pretrained model with a deep learning ......

Adding Custom Layers on Top of a Hugging Face Model

Learn how to extract the hidden states from a Hugging Face model body, modify/add task-specific layers on top of it and train the...

Fine-Tuning with Hugging Face Trainer - YouTube

In this tutorial I explain how I was using Hugging Face Trainer with PyTorch to fine - tune LayoutLMv2 model for data extraction...

Tutorial 2- Fine Tuning Pretrained Model On Custom Dataset ...

github: https://github.com/krishnaik06/Huggingfacetransformer In this tutorial, we will show you how to fine - tune a pretrained model from ...

BERT Fine-Tuning Tutorial with PyTorch - Chris McCormick

We'll be using BertForSequenceClassification. This is the normal BERT model with an added single linear layer on top for classification that we ...