`additional_special_tokens` is replaced instead of being updated

Reproduction

from transformers import T5Tokenizer
from transformers.tokenization_utils import AddedToken

tokenizer = T5Tokenizer.from_pretrained("t5-small", extra_ids=0, additional_special_tokens=["new_token_1"])

print(tokenizer.additional_special_tokens)
print(tokenizer.added_tokens_encoder)

tokenizer.add_special_tokens({"additional_special_tokens": ["new_token_2"]})
print(tokenizer.additional_special_tokens)
print(tokenizer.added_tokens_encoder)

tokenizer.add_special_tokens({"additional_special_tokens": ["new_token_3"]})
print(tokenizer.additional_special_tokens)
print(tokenizer.added_tokens_encoder)

gives

['new_token_1']
{'new_token_1': 32000}
['new_token_2']
{'new_token_1': 32000, 'new_token_2': 32001}
['new_token_3']
{'new_token_1': 32000, 'new_token_2': 32001, 'new_token_3': 32002}

Expected behavior

We should get

[‘new_token_1’, ‘new_token_2’, ‘new_token_3’]

[SaulLu]

To share what has been discussed on slack.

I think the naming is confusing but the current behavior of the method is useful because we need to be able to completely change the list of tokens associated with additional special tokens. Maybe calling this method set_special_tokens would be less confusing?

Basically, the difficulty here is that I think it’s not necessarily obvious what types of tokens a tokenizer can have.

A tokenizer has a vocabulary (a dictionary mapping tokens to ids) that consists of:

1. The vocabulary learned with a tokenization algorithm (BPE, wordpiece or Unigram)
2. The vocabulary corresponding to the tokens added afterwards and which will be isolated upstream of the tokenization algorithm. They are called added_tokens.

Afterwards, some tokens can be special in the sense that they will be associated with a specific role for the model (e.g. the begin of sentence token, the mask token). The additional special tokens list is here to be able to flag more tokens as special for the models that need them [Even if for the model it is not safe to go and find a special token based on an index in a list but that’s another subject].

[NielsRogge]

I also saw this issue, and found it very unintuitive. Should be addressed in V5 imo. Cc @LysandreJik @SaulLu