unk_id is missing for SentencepieceTokenizer

Trained sentencepiece tokenizer from Tokenizer library with some added tokens

 "added_tokens": [
    {
      "id": 0,
      "special": true,
      "content": "<unk>",
      "single_word": false,
      "lstrip": false,
      "rstrip": false,
      "normalized": false
    },
    {
      "id": 1,
      "special": true,
      "content": "<s>",
      "single_word": false,
      "lstrip": false,
      "rstrip": false,
      "normalized": false
    },
    {
      "id": 2,
      "special": true,
      "content": "</s>",
      "single_word": false,
      "lstrip": false,
      "rstrip": false,
      "normalized": false
    },
    {
      "id": 3,
      "special": true,
      "content": "<cls>",
      "single_word": false,
      "lstrip": false,
      "rstrip": false,
      "normalized": false
    },
    {
      "id": 4,
      "special": true,
      "content": "<sep>",
      "single_word": false,
      "lstrip": false,
      "rstrip": false,
      "normalized": false
    },
    {
      "id": 5,
      "special": true,
      "content": "<pad>",
      "single_word": false,
      "lstrip": false,
      "rstrip": false,
      "normalized": false
    },
    {
      "id": 6,
      "special": true,
      "content": "<mask>",
      "single_word": false,
      "lstrip": false,
      "rstrip": false,
      "normalized": false
    },
    {
      "id": 7,
      "special": true,
      "content": "<eod>",
      "single_word": false,
      "lstrip": false,
      "rstrip": false,
      "normalized": false
    },
    {
      "id": 8,
      "special": true,
      "content": "<eop>",
      "single_word": false,
      "lstrip": false,
      "rstrip": false,
      "normalized": false
    }
  ],

When this tokenizer encounters unknown words, it throws unk_id is missing error.

Exception: Encountered an unknown token but unk_id is missing How do I set unk_id for this tokenizer?

This is how I load my tokenizer

tokenizer = Tokenizer.from_file('./unigram.json')
tokenizer = XLNetTokenizerFast(tokenizer_object=tokenizer, unk_token="<unk>")

[darwinharianto]

Thanks! Manually changing unk_id to 0 make it works

[SaulLu]

Thanks for the document, I think the error comes from the fact that the value associated to the model key and the unk_id subkey in your unigram.json file is null instead of 0 ("<unk>" is the first token of your vocabulary).

If you have used the training script of SentencePieceUnigramTokenizer provided in 🤗 tokenizers library, I’ve opened a PR here to solve this missing information for the future trainings - as the unk_token need to be pass to the Trainer and currently you haven’t the opportunity to do it. 🙂

However, I guess you don’t want to re-train your tokenizer. In this case, the simplest is to change by hand the value in the unigram.json file associated to the unk_id key so that it matches the id of the unknown token (in your case 0).

I would be happy to know if this indeed solves the error you had ☺️