Roberta-large using BPE tokenizer generates multi tokens.

Roberta-large uses byte-level Byte-Pair-Encoding. It avoids the common PET training.

For example, Verbalization "Society" does not correspond to a single token, got ['Soc', 'iety']

Now I just comment the code assert ( # len(ids) == 1 in utils.py to enforce using the first tokenizer.

But I don’t know whether it will affect the accuracy. So is there any alternative since PET uses Roberta-large by default?

Thanks~

[huchinlp]

Hi,

GPT-2 and Roberta tokenizers will recognize the space before a word and replace it with a “Ġ”. Actually, “Society” is not a token in the vocab but “ĠSociety” is a valid one. You can call tokenizer.convert_tokens_to_ids("ĠSociety") and the result is 3930.

The only thing you need to do is replace “tokenizer.encode(xxxxx)” with the following lines:

if tokenizer.convert_tokens_to_ids(word) == tokenizer.unk_token_id:
     space_word = "Ġ" + word
     id = tokenizer.convert_tokens_to_ids(space_word)
else:
     id = tokenizer.convert_tokens_to_ids(word)

Refer to this thread for more details: https://discuss.huggingface.co/t/bpe-tokenizers-and-spaces-before-words/475?u=joaogante Best.

[nieallen]

Hi, how to training PET model uses xlm-roberta with byte-level Byte-Pair-Encoding?