Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

cannot import name AddedToken

See original GitHub issue

🐛 Bug

Information

Model I am using (Bert, XLNet …): Albert

Language I am using the model on (English, Chinese …): English

 27 from typing import List, Optional, Sequence, Tuple, Union
 28

—> 29 from tokenizers import AddedToken, Encoding 30 from tokenizers.decoders import Decoder 31 from tokenizers.implementations import BaseTokenizer

ImportError: cannot import name ‘AddedToken’

The problem arises when using:

the official example scripts: (give details below)
my own modified scripts: (give details below)

The tasks I am working on is:

an official GLUE/SQUaD task: (give the name):
my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

Expected behavior

Environment info

transformers version:
Platform:
Python version:
PyTorch version (GPU?):
Tensorflow version (GPU?):
Using GPU in script?:
Using distributed or parallel set-up in script?:

Issue Analytics

State:
Created 3 years ago
Comments:6

Top GitHub Comments

2reactions

wangxupengcommented, Apr 16, 2020

you need install transformers this way:

git clone https://github.com/huggingface/transformers
cd transformers
pip install .

1reaction

nrjvarshneycommented, Apr 8, 2020

Created another environment from scratch and it got resolved.

Top Results From Across the Web

ImportError caused by file with the same name in working dir ...

I think I've understood what causes the issue - it is shadowing file with the same name in package transformer (that internally import ......

Source code for transformers.tokenization_utils_base

class AddedToken(UserString): # """ AddedToken represents a token to be added to a Tokenizer # An AddedToken can have special options defining the...

Hugging Face on Twitter: "@DiogoCortiz Sorry about that. Is ...

Today I am getting the error "cannot import name 'BertWordPieceTokenizer'" when I do "from tokenizers import BertWordPieceTokenizer".

tokenizers - PyPI

Train new vocabularies and tokenize using 4 pre-made tokenizers (Bert WordPiece and the 3 most common BPE versions). Extremely fast (both training and ......

Source code for paddlenlp.transformers.tokenizer_utils

We can't do it directly within the loop because of how # python iteration works ... whose name ends with `_token`) as attributes...