Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Dependency version check fails for tokenizers

See original GitHub issue

Environment info

transformers version: 4.5.0
Platform: Linux-4.15.0-134-generic-x86_64-with-glibc2.10
Python version: 3.8.5
PyTorch version (GPU?): 1.8.1 (False)
Tensorflow version (GPU?): 2.4.0 (False)
Using GPU in script?: N/A
Using distributed or parallel set-up in script?: N/A
tokenizers version: 0.10.2 (checked also 0.10.1)

Who can help

@stas00, @sgugger

Information

When importing transformers, the new dependency version check code (#11061) seems to fail for the tokenizers library: importlib.metadata.version('tokenizers') returns None instead of the version string.

The problem arises when using:

the official example scripts: (give details below)
my own modified scripts: (give details below)

To reproduce

Steps to reproduce the behavior:

import transformers

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/guyrosin/miniconda3/envs/pt/lib/python3.8/site-packages/transformers/__init__.py", line 43, in <module>
    from . import dependency_versions_check
  File "/home/guyrosin/miniconda3/envs/pt/lib/python3.8/site-packages/transformers/dependency_versions_check.py", line 41, in <module>
    require_version_core(deps[pkg])
  File "/home/guyrosin/miniconda3/envs/pt/lib/python3.8/site-packages/transformers/utils/versions.py", line 101, in require_version_core
    return require_version(requirement, hint)
  File "/home/guyrosin/miniconda3/envs/pt/lib/python3.8/site-packages/transformers/utils/versions.py", line 92, in require_version
    if want_ver is not None and not ops[op](version.parse(got_ver), version.parse(want_ver)):
  File "/home/guyrosin/miniconda3/envs/pt/lib/python3.8/site-packages/packaging/version.py", line 57, in parse
    return Version(version)
  File "/home/guyrosin/miniconda3/envs/pt/lib/python3.8/site-packages/packaging/version.py", line 296, in __init__
    match = self._regex.search(version)
TypeError: expected string or bytes-like object

The root problem is this:

from importlib.metadata import version
version('tokenizers') # returns None

Expected behavior

importlib.metadata.version('tokenizers') should return its version string.

Issue Analytics

State:
Created 2 years ago
Comments:6 (6 by maintainers)

Top GitHub Comments

1reaction

stas00commented, Apr 7, 2021

Yay! Glad it worked, @guyrosin!

1reaction

guyrosincommented, Apr 7, 2021

OK, it seems like my environment was corrupted indeed - there was a tokenizers-0.9.4.dist-info folder inside my env’s site-packages folder… After deleting it (manually) and reinstalling tokenizers, everything works! Thanks a lot for your help @stas00!

Top Results From Across the Web

Installation Error - Failed building wheel for tokenizers #2831

1 by manually install the last version of tokenizers (0.6.0) instead of 0.5.2 that is required in the transformer package. pip install ...

Could not build wheels for tokenizers, which is required to ...

I got the same error when I tried to install the transformers library and I was using python 3.6 but when I upgraded...

ERROR: Could not build wheels for tokenizers in Python

To solve the error "Could not build wheels for tokenizers", run the pip install --upgrade pip command to upgrade your pip version, ...

Migrating from previous packages

This means that the tokenizers that depend on the SentencePiece library will not be available with a standard transformers installation. This includes the...

Install spaCy · spaCy Usage Documentation

spaCy is a free open-source library for Natural Language Processing in Python. It features NER, POS tagging, dependency parsing, word vectors and more....