Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Can't load tokenizer for longt5-xl

See original GitHub issue

System Info

transformers version: 4.20.0 Platform: Linux-4.15.0-135-generic Python version: 3.8.13 PyTorch version (GPU?): torch==1.10.2+cu113 Using GPU in script?: no Using distributed or parallel set-up in script?: no

Who can help?

@patrickvonplate

Information

The official example scripts
My own modified scripts

Tasks

An officially supported task in the examples folder (such as GLUE/SQuAD, …)
My own task or dataset (give details below)

Reproduction

I’m using the example provided on this page: https://huggingface.co/google/long-t5-tglobal-xl:


from transformers import AutoTokenizer, LongT5Model

tokenizer = AutoTokenizer.from_pretrained("google/long-t5-tglobal-xl")

Expected behavior

A tokenizer for a longt5-xl works

Issue Analytics

State:
Created a year ago
Reactions:1
Comments:5 (4 by maintainers)

Top GitHub Comments

1reaction

LysandreJikcommented, Aug 1, 2022

Sounds good, I’ll take care of that. Thanks!

0reactions

LysandreJikcommented, Aug 1, 2022

Should work now, this was the only repository that needed to be updated (was lacking the tokenizer files). Feel free to close this issue if your problem is solved!

Top Results From Across the Web

What to do when HuggingFace throws "Can't load tokenizer"

If you were trying to load it from 'Models - Hugging Face', make sure you don't have a local directory with the same...

OSError: Can't load tokenizer - Stack Overflow

I want to train an XLNET language model from scratch. First, I have trained a tokenizer as follows:

Can't load a saved tokenizer with AutoTokenizer ... - GitHub

Issue. # On Jupyter Notebook.exe tokenizer = AutoTokenizer. from_pretrained('prajjwal1/bert-tiny') # Output of the cell Could not locate the ...

Can't load tokenizer : r/KoboldAI - Reddit

What does it mean if KAI can't load a tokenizer for my finetune of GPT-2? It's weird it can't find it. In the...

spaCy 101: Everything you need to know

spaCy is a free, open-source library for advanced Natural Language Processing ... Features; Linguistic annotations; Tokenization; POS tags and dependencies ...