Adding special tokens to the model

Hello, I am trying to use model.tokenizer.add_special_tokens(special_tokens_dict) to add some special tokens to the model. But after doing that i received indexing error (IndexError: index out of range in self ) when i wanted to encode a sentence. I wonder to know how i can learn the vector representations of new tokens? something like model.resize_token_embeddings(len(t))

[nreimers]

You can use this code:

tokens = ["TOK1", "TOK2"]
word_embedding_model = model._first_module()   #Your models.Transformer object
word_embedding_model.tokenizer.add_tokens(tokens, special_tokens=True)
word_embedding_model.auto_model.resize_token_embeddings(len(word_embedding_model.tokenizer))

[nreimers]

Yes, it is correct.