`additional_special_tokens` is replaced instead of being updated
See original GitHub issueReproduction
from transformers import T5Tokenizer
from transformers.tokenization_utils import AddedToken
tokenizer = T5Tokenizer.from_pretrained("t5-small", extra_ids=0, additional_special_tokens=["new_token_1"])
print(tokenizer.additional_special_tokens)
print(tokenizer.added_tokens_encoder)
tokenizer.add_special_tokens({"additional_special_tokens": ["new_token_2"]})
print(tokenizer.additional_special_tokens)
print(tokenizer.added_tokens_encoder)
tokenizer.add_special_tokens({"additional_special_tokens": ["new_token_3"]})
print(tokenizer.additional_special_tokens)
print(tokenizer.added_tokens_encoder)
gives
['new_token_1']
{'new_token_1': 32000}
['new_token_2']
{'new_token_1': 32000, 'new_token_2': 32001}
['new_token_3']
{'new_token_1': 32000, 'new_token_2': 32001, 'new_token_3': 32002}
Expected behavior
We should get
[‘new_token_1’, ‘new_token_2’, ‘new_token_3’]
Issue Analytics
- State:
- Created 10 months ago
- Comments:6 (4 by maintainers)
Top Results From Across the Web
`additional_special_tokens` is replaced instead of being ...
I think the naming is confusing but the current behavior of the method is useful because we need to be able to completely...
Read more >Utilities for Tokenizers - Hugging Face
If special tokens are NOT in the vocabulary, they are added to it (indexed starting from the last index of the current vocabulary)....
Read more >How to add new special token to the tokenizer? - Stack Overflow
1 Answer 1 · i have added [EOT] token to the tokenizer using add_tokens. then i added [EOT] in data after every turn....
Read more >Use Tokens in Job Steps - SQL Server Agent | Microsoft Learn
Updating Job Steps to Use Macros ; ESCAPE macro used, All tokens in jobs are successfully replaced. Tokens activated by alerts are not...
Read more >Replace a Token for a User - RSA Community - 629587
You can assign a replacement token to a user if a user's token has been permanently lost or destroyed, or if the current...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
To share what has been discussed on slack.
I think the naming is confusing but the current behavior of the method is useful because we need to be able to completely change the list of tokens associated with additional special tokens. Maybe calling this method
set_special_tokens
would be less confusing?Basically, the difficulty here is that I think it’s not necessarily obvious what types of tokens a tokenizer can have.
A tokenizer has a vocabulary (a dictionary mapping tokens to ids) that consists of:
added_tokens
.Afterwards, some tokens can be special in the sense that they will be associated with a specific role for the model (e.g. the begin of sentence token, the mask token). The additional special tokens list is here to be able to flag more tokens as special for the models that need them [Even if for the model it is not safe to go and find a special token based on an index in a list but that’s another subject].
I also saw this issue, and found it very unintuitive. Should be addressed in V5 imo. Cc @LysandreJik @SaulLu