error loading facebook/opt-30b with text generation pipeline using 8bit mixed precision

System Info

• transformers version: 4.24.0
• Platform: Linux-5.4.0-109-generic-x86_64-with-glibc2.10
• Python version: 3.8.13
• Huggingface_hub version: 0.11.0
• PyTorch version (GPU?): 1.13.0a0+d0d6b1f (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: yes
• Using distributed or parallel set-up in script?: no

Who can help?

@patrickvonplaten, @Narsil, @gante

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, …)
• My own task or dataset (give details below)

Reproduction

Running the following on a system with one (NVIDIA A5000) GPU:

from transformers import pipeline
model = "facebook/opt-30b"
model_kwargs = {"device_map": "auto", "load_in_8bit": True}
generator = pipeline(task="text-generation", model=model, device=0, model_kwargs=model_kwargs)

yields error: ValueError: Could not load model facebook/opt-30b with any of the following classes: (<class 'transformers.models.auto.modeling_auto.AutoModelForCausalLM'>, <class 'transformers.models.opt.modeling_opt.OPTForCausalLM'>).

Expected behavior

Should be able to create generator with no problem and generate text with generator.__call__.

The code works with no error when using smaller opt model checkpoints: “facebook/opt-2.7b”, “facebook/opt-6.7b”.

Can create model, tokenizer, and generate without pipeline using AutoModelForCausalLM.from_pretrained(model, device_map="auto") with model="facebook/opt-30b" despite the error message.

[morrisalp]

Thanks! I mainly wanted to see what the largest LLM I could fit on one of my GPUs would be using mixed precision, and I couldn’t tell previously if the 30B model would be OOM due to the other errors…

[sgugger]

Please provide the full traceback, as we can’t see what’s happening otherwise especially since I can’t reproduce locally on my side. cc @younesbelkada who might have better luck reproducing the bug!