Hugging Face GPT-2 Tokenizer

Hello,

I know that if I choose to add any new “special token” onto the pre-made GPT-2 tokenizer, and if I want to use the pre-trained GPT-2 model for my analysis, I will need to re-train the pre-trained GPT-2 to make the model learn that new special token.

But what if I just add an extra non-special token? for example, a word “paradox” is not included in the existing GPT-2 tokenizer, so say I add the word “paradox” to the existing set of GPT-2 vocabulary, like below:

# load the pre-trained GPT2-tokenizer
gpt2_tokenizer = GPT2Tokenizer.from_pretrained('gpt2')

# adding a new word (not special token) to the existing vocabulary,
# but I am not making any changes to the pre-assigned special tokens
gpt2_tokenizer.add_tokens("paradox")

# get the pre-trained HuggingFace GPT2DoubleHeadsModel 
model_gpt2DoubleHeadsModel = GPT2DoubleHeadsModel.from_pretrained('gpt2', output_hidden_states = True)

# resize the token embeddings
# (not sure what this function does)
model_gpt2DoubleHeadsModel.resize_token_embeddings(len(gpt2_tokenizer))

Given that I didn’t make any changes to the special tokens in the GPT-2-tokenizer, do I still need to train the already pre-trained GPT2DoubleHeadsModel before I start using it, just because I added a new word to the set of vocabulary?

Thank you,

[LysandreJik]

You would need to fine-tune your GPT-2 model on a dataset containing the word, yes. The reason being that your model needs to understand in which context is the word used, what it means, etc.

[h56cho]

Thank you! I was able to confirm that what you mentioned in your previous post also works for my case.