Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Customize tokenizer in model card's widget

See original GitHub issue

I trained a Chinese Roberta model. In the model card, the widget uses a tokenizer defined in config.json(RobertaTokenizer). But my model uses BertTokenizer. Can I customize the tokenizer in the widget of the model card just like I can choose any combination of model and tokenizer in a pipeline?

Issue Analytics

State:
Created 3 years ago
Comments:5 (5 by maintainers)

Top GitHub Comments

3reactions

julien-ccommented, Oct 29, 2020

Yes, this is possible. See https://github.com/huggingface/transformers/commit/ed71c21d6afcbfa2d8e5bb03acbb88ae0e0ea56a, you should add a tokenizer_class attribute to your config.json with the tokenizer class you want to use.

cc @sgugger @LysandreJik I have no idea if this is currently documented or just in the code 🤭

2reactions

julien-ccommented, Oct 29, 2020

arg, who does that guy think he is? 😂

Top Results From Across the Web

Customize tokenizer in model card's widget - Beginners

I trained a Chinese Roberta model. In the model card, the widget uses a tokenizer defined in config.json( RobertaTokenizer ).

Create a Tokenizer and Train a Huggingface RoBERTa Model ...

Create and train a byte-level, Byte-pair encoding tokenizer with the same special tokens as RoBERTa; Train a RoBERTa model from scratch using ...

Basis Theory - Quickly tokenize anything

A compliant and developer-friendly data tokenization and encryption ... Customize the length and format of your tokens using the Liquid syntax, Expressions.

Digital Wallets and Tokenization | Marqeta Docs

Digital Wallets and Tokenization. A digital wallet is a device or system for storing digitized versions of payment cards.

Tokenization payment technology guide - Adyen

Does payment tokenization suit my business? Tokenization is suited to any businesses with subscription- based business models or which generate significant ...