How can I generate sentencepiece file or vocabulary from tokenizers?

After I make custom tokenizer using Tokenizers library, I could load it into XLNetTokenizerFast using

tokenizer = Tokenizer.from_file("unigram.json")
tok = XLNetTokenizerFast(tokenizer_object=tokenizer)

After I called

tok .save_vocabulary("ss")

it throws an error since I didnt load XLNetTokenizerFast using spm file. I believe save_vocabulary is looking for vocab_file parameter.

Is there any way to save_vocab after loading it from XLNetTokenizer?

[SaulLu]

Thank you very much for reporting this problem @darwinharianto .

I have the impression that it linked to the problem reported in this issue and that I had started to work on in this PR. As the cleanest looking fix requires quite a bit of work, I had put it on hold. I’ll try to work on it again at the beginning of the week.

[SaulLu]

Duly noted! If it’s to do a sanity check, would it be ok to compare the files of the fast version of the tokenizers (in particular the tokenizer.json file)?

To retrieve these files for the new tokenizer tok you made, the following command should work:

tok.save_pretrained("./dump", legacy_format=False)

For information, the vocabulary will be visible in the tokenizer.json file.