Cannot load reformer-enwik8 tokenizer

🐛 Bug

Information

Model I am using (Bert, XLNet …): Reformer tokenizer

To reproduce

Steps to reproduce the behavior:

1. Try to load the pretrained reformer-enwik8 tokenizer with AutoTokenizer.from_pretrained("google/reformer-enwik8")

This is the error I got:

OSError                                   Traceback (most recent call last)
<ipython-input-51-ab9a64363cc0> in <module>
----> 1 AutoTokenizer.from_pretrained("google/reformer-enwik8")

~/.virtualenvs/sparseref/lib/python3.7/site-packages/transformers-2.9.0-py3.7.egg/transformers/tokenization_auto.py in from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
    198                     return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
    199                 else:
--> 200                     return tokenizer_class_py.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
    201 
    202         raise ValueError(

~/.virtualenvs/sparseref/lib/python3.7/site-packages/transformers-2.9.0-py3.7.egg/transformers/tokenization_utils.py in from_pretrained(cls, *inputs, **kwargs)
    896 
    897         """
--> 898         return cls._from_pretrained(*inputs, **kwargs)
    899 
    900     @classmethod

~/.virtualenvs/sparseref/lib/python3.7/site-packages/transformers-2.9.0-py3.7.egg/transformers/tokenization_utils.py in _from_pretrained(cls, pretrained_model_name_or_path, *init_inputs, **kwargs)
   1001                     ", ".join(s3_models),
   1002                     pretrained_model_name_or_path,
-> 1003                     list(cls.vocab_files_names.values()),
   1004                 )
   1005             )

OSError: Model name 'google/reformer-enwik8' was not found in tokenizers model name list (google/reformer-crime-and-punishment). We assumed 'google/reformer-enwik8' was a path, a model identifier, or url to a directory containing vocabulary files named ['spiece.model'] but couldn't find such vocabulary files at this path or url.

I tried with and without google/, same result. However, it did print the download progress bar. Trying to load the crime-and-punishment Reformer tokenizer works.

• transformers version: 2.9.0
• Platform: macOS
• Python version: 3.7
• PyTorch version (GPU?): 1.4.0, no GPU
• Using distributed or parallel set-up in script?: no

[bratao]

@erickrf can you share how you got to train the “reformer” model. I´m trying to utilize the “google/reformer-enwik8” to train a Portuguese model but I just got the same error of Model name 'google/reformer-enwik8' was not found in tokenizers

[BramVanroy]

@LeopoldACC Please post a new issue so that some one can have a look.