Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Unable to load DeBERTa-v3 tokenizer

See original GitHub issue

from transformers import AutoTokenizer
  
tokenizer = AutoTokenizer.from_pretrained("microsoft/mdeberta-v3-base")

Gives me an error: ValueError: This tokenizer cannot be instantiated. Please make sure you have sentencepiece installed in order to use this tokenizer. Installing sentencepiece doesn’t help.

Issue Analytics

State:
Created 2 years ago
Comments:5 (3 by maintainers)

Top GitHub Comments

6reactions

LysandreJikcommented, Nov 24, 2021

If you’re using colab, then have you restarted the runtime after installing sentencepiece?

1reaction

maiiabocharovacommented, Nov 24, 2021

If you’re using colab, then have you restarted the runtime after installing sentencepiece?

Thank you!!! It solved the issue. Thank you once again!

Top Results From Across the Web

Cant load deberta tokenizer - Beginners - Hugging Face Forums

I'm running transformers version 3.0.0. AutoModel.from_pretrained('microsoft/deberta-base') run fine for me but the autotokenizer fails:.

debertav3 baseline - Kaggle

... #can't load #model_name = "plncmm/deberta-clinical-scratch-cowese-es" ... In practice this means that the fast version of the tokenizer can produce ...

DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre ...

This paper presents a new pre-trained language model, DeBERTaV3, which improves the original DeBERTa model by replacing mask language modeling ( ...

Performance of Multiple Pretrained BERT Models to Automate ...

This figure demonstrates the process of “tokenization,” an automatic ... that each of the algorithms in this study failed to recognize the words...

ML and NLP Research Highlights of 2021 - Sebastian Ruder

In mathematics, ML was shown to be able to guide the intuition of ... of tokenized subwords has become the standard input format...