OpenOnDemandZipFile fail to open punkt.zip on python3.x

An exception raises when excuting an simple tutorial in nltk_book as the following command. $ python3 -c 'from nltk import word_tokenize; text = word_tokenize("And now for something completely different")'

I think the problem is caused by the decorator @py3_data on ZipFilePathPointer.__init__ and OpenOnDemandZipFile.__init__. This decorator will append ‘/PY3’ to the first arg, which is a a zip filepath as str, like “~/nltk_data/tokenizers/punkt.zip”, but “~/nltk_data/tokenizers/punkt.zip/PY3” can’t be opened as a zipfile.

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/git/nltk/nltk/tokenize/__init__.py", line 109, in word_tokenize
    return [token for sent in sent_tokenize(text, language)
  File "/git/nltk/nltk/tokenize/__init__.py", line 93, in sent_tokenize
    tokenizer = load('tokenizers/punkt/{0}.pickle'.format(language))
  File "/git/nltk/nltk/data.py", line 808, in load
    opened_resource = _open(resource_url)
  File "/git/nltk/nltk/data.py", line 926, in _open
    return find(path_, path + ['']).open()
  File "/git/nltk/nltk/data.py", line 648, in find
    raise LookupError(resource_not_found)
LookupError: 
**********************************************************************
  Resource 'tokenizers/punkt/PY3/english.pickle' not found.
  Please use the NLTK Downloader to obtain the resource:  >>>
  nltk.download()
  Searched in:
    - '/home/joybin/nltk_data'
    - '/usr/share/nltk_data'
    - '/usr/local/share/nltk_data'
    - '/usr/lib/nltk_data'
    - '/usr/local/lib/nltk_data'
    - ''
**********************************************************************

[stevenbird]

@alvations fyi in compat.py:

# The following datasets have a /PY3 subdirectory containing
# a full copy of the data which has been re-encoded or repickled.
DATA_UPDATES = [("chunkers", "maxent_ne_chunker"),
                ("help", "tagsets"),
                ("taggers", "maxent_treebank_pos_tagger"),
                ("tokenizers", "punkt")]

From memory these extra subdirectories were created manually.

[joybinchen]

@alvations I’m using python3.5 on ubuntu 17.04. I’m sure I have downloaded all zip file under ~/nltk_data. I can run the case without any Exception when ~/nltk_data/tokenizers/punkt.zip is unpacked.

$ python3 -c "import sys; print (sys.version)"
3.5.3 (default, Jan 19 2017, 14:11:04) 
[GCC 6.3.0 20170118]

$ python -m nltk.downloader punkt
/usr/lib/python3.5/runpy.py:125: RuntimeWarning: 'nltk.downloader' found in sys.modules after import of package 'nltk', but prior to execution of 'nltk.downloader'; this may result in unpredictable behaviour
  warn(RuntimeWarning(msg))
[nltk_data] Downloading package punkt to /home/joybin/nltk_data...
[nltk_data]   Package punkt is already up-to-date!