01_how-to-train.ipynb broken

🐛 Bug

To reproduce

Steps to reproduce the behavior:

1. Go to https://github.com/huggingface/transformers/tree/master/examples
2. Click the colab for language-modeling: https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb
3. Run notebook

Expected behavior

The notebook finishes succesfuly

What I get is:

---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-5-52625a7c86e5> in <module>()
      1 get_ipython().system('mkdir EsperBERTo')
----> 2 tokenizer.save("EsperBERTo")

/usr/local/lib/python3.6/dist-packages/tokenizers/implementations/base_tokenizer.py in save(self, path, pretty)
    330                 A path to the destination Tokenizer file
    331         """
--> 332         return self._tokenizer.save(path, pretty)
    333 
    334     def to_str(self, pretty: bool = False):

Exception: Is a directory (os error 21)

Environment info

• transformers version: 2.11.0
• Platform: Linux-4.19.104±x86_64-with-Ubuntu-18.04-bionic
• Python version: 3.6.9
• PyTorch version (GPU?): 1.5.0+cu101 (True)
• Tensorflow version (GPU?): not installed (NA)
• Using GPU in script?: NA
• Using distributed or parallel set-up in script?: NA

[n1t0]

Yes, tokenizers 0.8.0 introduces the full tokenizer serialization, whereas before it saved the “model” only (vocab.json + merges.txt for BPE). So the save method should be used like that: .save("tokenizer.json") and it saves the entire tokenizer to a JSON file. We need to update the Notebook to use this new serialization method, but in the meantime, the only thing needed to make it work exactly like before is to replace:

!mkdir EsperBERTo
tokenizer.save("EsperBERTo")

by

!mkdir EsperBERTo
tokenizer.save_model("EsperBERTo")

[thomwolf]

This one is for me (this method was actually not working as intended under the hood for Fast-tokenizers…)