Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

importing torchtext: module 'tokenize' has no attribute 'open'

See original GitHub issue

When trying to import any module within torchtext (either interactively or in a script), I get the following stack trace:

>>> import torchtext
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/torchtext/__init__.py", line 1, in <module>
    from . import data
  File "/usr/local/lib/python3.6/dist-packages/torchtext/data/__init__.py", line 1, in <module>
    from .batch import Batch
  File "/usr/local/lib/python3.6/dist-packages/torchtext/data/batch.py", line 1, in <module>
    import torch
  File "/usr/local/lib/python3.6/dist-packages/torch/__init__.py", line 244, in <module>
    _C._initExtension(manager_path())
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 126, in <module>
    _lazy_call(_check_capability)
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 124, in _lazy_call
    _queued_calls.append((callable, traceback.format_stack()))
  File "/usr/lib/python3.6/traceback.py", line 193, in format_stack
    return format_list(extract_stack(f, limit=limit))
  File "/usr/lib/python3.6/traceback.py", line 207, in extract_stack
    stack = StackSummary.extract(walk_stack(f), limit=limit)
  File "/usr/lib/python3.6/traceback.py", line 358, in extract
    f.line
  File "/usr/lib/python3.6/traceback.py", line 282, in line
    self._line = linecache.getline(self.filename, self.lineno).strip()
  File "/usr/lib/python3.6/linecache.py", line 16, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib/python3.6/linecache.py", line 47, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib/python3.6/linecache.py", line 136, in updatecache
    with tokenize.open(fullname) as fp:
AttributeError: module 'tokenize' has no attribute 'open'

This happens with the following relevant software:

Ubuntu 18.04
Python 3.6.5
torch 0.4.0
torchtext 0.2.3

Issue Analytics

State:
Created 5 years ago
Comments:11 (3 by maintainers)

Top GitHub Comments

13reactions

JesseTGcommented, Jul 9, 2018

…ah, I think I’ve figured it out. It turns out I was in a directory with a Python module in my project called “tokenize”, and that module was being preferred over another one. I renamed it, and now this works fine.

1reaction

Xie-Minghuicommented, Feb 2, 2021

I encountered this problem too. tokenize.py is a file of the same name of python library file. When you debug, python debugger mistask your file as a standard library file.Therefore you can change your file name from tokenize.py to tokenize_.py. After that everything goes ok.

Top Results From Across the Web

AttributeError: module 'tokenize' has no attribute 'open'

I don't know much about python but I will try to answer your question. open up command prompt and type py -m pip...

AttributeError: module 'tokenize' has no attribute 'open' - Reddit

AttributeError: module 'tokenize' has no attribute 'open'. All i am trying to do is turn a .py into a single file .exe, i...

module 'torchtext.data' has no attribute 'field' - You.com

For the users who still want the legacy components, they can add legacy to the import path. Try it with ENGLISH = torchtext.legacy.data.field(tokenize= ......

torchtext - PyPI

examples: Example NLP workflows with PyTorch and torchtext library. Installation. We recommend Anaconda as a Python package management system. Please refer to ...

Source code for torchtext.data.utils - PyTorch

import random import re from contextlib import contextmanager from copy import ... default tokenizer is string.split(), added as a module function for ...