Model download broken after removal of use_auth_token

Describe the bug

After updating my git checkout to a version that removed use_auth_token parameters, diffusers cannot download CompVis/stable-diffusion-v1-4 anymore with an error 401.

I am logged in with the same auth token as before. Downloading another model that is not protected with an auth token using the same script works.

Another possibly related issue: Using the different model from huggingface, the script re-downloaded it even when no new version was published. As far as I understand the cache structure, even a change in the snapshot id for some reason should still be able to avoid the ~4GB download of the model.

Reproduction

Run

python diffusers/examples/textual_inversion/textual_inversion.py   --pretrained_model_name_or_path=CompVis/stable-diffusion-v1-4 (...)

or an inference script, that had use_auth_token removed.

Logs

Traceback (most recent call last):
  File "stable-diffusion/venv/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1744, in from_pretrained
    resolved_vocab_files[file_id] = cached_path(
  File "stable-diffusion/venv/lib/python3.10/site-packages/transformers/utils/hub.py", line 284, in cached_path
    output_path = get_from_cache(
  File "stable-diffusion/venv/lib/python3.10/site-packages/transformers/utils/hub.py", line 502, in get_from_cache
    _raise_for_status(r)
  File "stable-diffusion/venv/lib/python3.10/site-packages/transformers/utils/hub.py", line 417, in _raise_for_status
    raise RepositoryNotFoundError(
transformers.utils.hub.RepositoryNotFoundError: 401 Client Error: Repository not found for url: https://huggingface.co/CompVis/stable-diffusion-v1-4/resolve/main/tokenizer/vocab.json. If the repo is private, make sure you are authenticated.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "stable-diffusion/diffusers/examples/textual_inversion/textual_inversion.py", line 591, in <module>
    main()
  File "stable-diffusion/diffusers/examples/textual_inversion/textual_inversion.py", line 372, in main
    tokenizer = CLIPTokenizer.from_pretrained(args.pretrained_model_name_or_path, subfolder="tokenizer")
  File "stable-diffusion/venv/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1762, in from_pretrained
    raise EnvironmentError(
OSError: CompVis/stable-diffusion-v1-4 is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo with `use_auth_token` or log in with `huggingface-cli login` and pass `use_auth_token=True`.

System Info

• diffusers version: 0.5.0.dev0
• Platform: Linux-5.19.0
• Python version: 3.10.7
• PyTorch version (GPU?): 1.12.1+cu116 (True)
• Huggingface_hub version: 0.9.1
• Transformers version: 4.21.2
• Using GPU in script?: yes
• Using distributed or parallel set-up in script?: no

[allo-]

Installing diffusers stable and then reinstalling the git version pulled a new huggingface-hub version and now the downloads work. I guess I need to install the requirements.txt more often between git updates.

[virtualramblas]

pip install --upgrade diffusers solved it for me.

This worked for me too on MacOS, Silicon M1.