Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

XLNetTokenizer throwing exception

See original GitHub issue

Hi,

I am trying to run xlnet and getting and exception at the initialization of XLNetTokenizer. I have following files under xlnet>base-cased directory:

config.json
pytorch_model.bin
spiece.model

when i run

python run_xlnet.py --do_test

I get the exception:

not a string

I have narrowed down the problem to the following line of code:

self.tokenizer = XLNetTokenizer(vocab_path, do_lower_case)

@lonePatient any help in this regard is much appreciated. Thank you

Issue Analytics

State:
Created 4 years ago
Comments:7 (3 by maintainers)

Top GitHub Comments

1reaction

lonePatientcommented, Feb 27, 2020

@AbdulRafay hi, I konw, convert vocab_path to str(vocan_path) ,and test_data = list(zip(test_data)

0reactions

AbdulRafaycommented, Feb 27, 2020

@lonePatient sorry i did not see it there 😃 Thanks alot for your help. I will close this issue now.

Top Results From Across the Web

Pretrained XLNetTokenizer not returning tokenizer #10797

Such an error: ImportError: XLNetTokenizer requires the SentencePiece library but it was not found in your environment. Checkout the ...

XLNet - Hugging Face

Construct a “fast” XLNet tokenizer (backed by HuggingFace's tokenizers library). Based on Unigram. This tokenizer inherits from PreTrainedTokenizerFast ...

XLNet Fine-Tuning Tutorial with PyTorch - Chris McCormick

We will use the raw version because we need to use the XLNet tokenizer to break the text down into tokens and chunks...

Data — Texar-PyTorch v0.1

XLNetTokenizer (pretrained_model_name=None, cache_dir=None, ... moving tensors to CUDA devices within this method would result in CUDA errors being thrown.

Don't throw exceptions in C#. Do this instead - YouTube

Check out my new "Integration testing in ASP .NET Core" course and use code INTESTING1 for 20% off (first 300): ...