Deberta Tokenizer convert_ids_to_tokens() is not giving expected results

Environment info

• transformers version: 4.3.0
• Platform: Colab
• Python version: 3.9
• PyTorch version (GPU?): No
• Tensorflow version (GPU?): No
• Using GPU in script?: No
• Using distributed or parallel set-up in script?: No

Information

I am using Deberta Tokenizer. convert_ids_to_tokens() of the tokenizer is not working fine.

The problem arises when using:

• my own modified scripts: (give details below)

The tasks I am working on is:

• an official GLUE/SQUaD task: (give the name)
• my own task or dataset

To reproduce

Steps to reproduce the behavior:

1. Get Debrta Tokenizer

from transformers import DebertaTokenizer
deberta_tokenizer = DebertaTokenizer.from_pretrained('microsoft/deberta-base')

1. Encode Some Example Using Tokenizer

example = "Hi I am Bhadresh. I found an issue in Deberta Tokenizer"
encoded_example = distilbert_tokenizer.encode(example)

1. Convert Ids to tokens:

distilbert_tokenizer.convert_ids_to_tokens(encoded_example )
"""
Output: ['[CLS]', '17250', '314', '716', '16581', '324', '3447', '13', '314', '1043', '281', '2071', '287', '1024', '4835', '64', '29130', '7509', '[SEP]']
"""

Colab Link For Reproducing

Expected behavior

It should return some tokens like this

['[CLS]', 'hi', 'i', 'am', 'b', '##had', '##resh', '.', 'i', 'found', 'an', 'issue', 'in', 'de', '##bert', '##a', 'token', '##izer', '[SEP]']

Not just convert an integer to string like the current behavior

Tagging SMEs for help:

@n1t0, @LysandreJik

[cronoik]

Yes. @LysandreJik

[cronoik]

You can convert them back with the following code:

from transformers import DebertaTokenizer
t = DebertaTokenizer.from_pretrained('microsoft/deberta-base')
example = "Hi I am Bhadresh. I found an issue in Deberta Tokenizer"

encoded_example = t.encode(example)

[t.gpt2_tokenizer.decode([t.gpt2_tokenizer.sym(id)]) if t.gpt2_tokenizer.sym(id) not in t.all_special_tokens else t.gpt2_tokenizer.sym(id) for id in encoded_example]

Output:

['[CLS]',
 'Hi',
 ' I',
 ' am',
 ' Bh',
 'ad',
 'resh',
 '.',
 ' I',
 ' found',
 ' an',
 ' issue',
 ' in',
 ' De',
 'bert',
 'a',
 ' Token',
 'izer',
 '[SEP]']

After some digging into the code, I am actually not sure if I should create a patch for it or not. I think with a patch we can probably also remove the method download_asset and refactor the load_vocab method.

I am not sure if this was discussed before but when we create the required files from the bpe_encoder.bin, we could probably get rid of the GPT2Tokenizer class in tokenization_deberta.py and the DebertaTokenizer could inherit directly from the GPT2Tokenizer (like the RobertaTokenizer).

I will leave it to @LysandreJik and @BigBird01 to decide what to do with it.