Changes ~ 14 Nov 2022 have broken stablediffusionimg2img pipeline (at least mps, maybe more)

Describe the bug

A very recent update to diffusers has broken the stable diffusion img2img pipeline with the libs and archs below (mps). Have not yet tested other architectures. Likely from changes in pipeline_stable_diffusion_img2img.py .

I’m using nightly torch updates. I rolled torch back from 14 Nov to 12 Nov (when I think the combination of diffusers and torch was working) and did not fix the problem.

Thank you for your great work.

Reproduction

core_prompt = “tiny minature brown circular monsters” negative_prompt = “” mode = “img” # img or text
theimg = “InputImg/ihc.png” import sys import diffusers from diffusers import DiffusionPipeline from diffusers import StableDiffusionImg2ImgPipeline import torch from torch import autocast device = “mps” import platform print(“Diffusers:”, diffusers.version) print(“Torch:”, torch.version) print(“Python:”, sys.version) print(“Platform:”, platform.version()) print(“Platform Release:”, platform.release()) stablediffusion = “CompVis/stable-diffusion-v1-4” ##stablediffusion = “stable-diffusion-v1-5”
print(“device:”, device) torch.set_num_threads(7) import random import pandas import time import os import PIL from PIL import Image from PIL.PngImagePlugin import PngInfo

img = Image.open(theimg) pipe = StableDiffusionImg2ImgPipeline.from_pretrained( stablediffusion, torch_dtype=torch.float32, revision=“fp32” ) pipe.to(device)

_ = pipe(core_prompt, init_image = img, strength = 0.5, num_inference_steps=1 )

Logs

Traceback (most recent call last):
  File "sd.new.py", line 160, in <module>
    _ =  pipe(prompt, init_image = img, strength = 0.5, num_inference_steps=1 )
  File "/opt/homebrew/Caskroom/miniconda/base/envs/ldm4/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/Users/nathan/Documents/R/SD.diffusers/diffusers/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_img2img.py", line 546, in __call__
    image = self.decode_latents(latents)
  File "/Users/nathan/Documents/R/SD.diffusers/diffusers/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_img2img.py", line 335, in decode_latents
    image = self.vae.decode(latents).sample
  File "/Users/nathan/Documents/R/SD.diffusers/diffusers/src/diffusers/models/vae.py", line 581, in decode
    dec = self.decoder(z)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/ldm4/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1423, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/nathan/Documents/R/SD.diffusers/diffusers/src/diffusers/models/vae.py", line 213, in forward
    sample = self.mid_block(sample)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/ldm4/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1423, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/nathan/Documents/R/SD.diffusers/diffusers/src/diffusers/models/unet_2d_blocks.py", line 301, in forward
    hidden_states = attn(hidden_states)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/ldm4/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1423, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/nathan/Documents/R/SD.diffusers/diffusers/src/diffusers/models/attention.py", line 288, in forward
    query_states = self.transpose_for_scores(query_proj)
  File "/Users/nathan/Documents/R/SD.diffusers/diffusers/src/diffusers/models/attention.py", line 270, in transpose_for_scores
    new_projection = projection.view(new_projection_shape).permute(0, 2, 1, 3)
RuntimeError: cannot reshape tensor of 0 elements into shape [0, 4096, 1, -1] because the unspecified dimension size -1 can be any value and is ambiguous

System Info

Diffusers: 0.8.0.dev0 Torch: 1.14.0.dev20221115 Python: 3.8.13 (default, Mar 28 2022, 06:13:39) [Clang 12.0.0 ] Platform: Darwin Kernel Version 22.1.0: Sun Oct 9 20:15:09 PDT 2022; root:xnu-8792.41.9~2/RELEASE_ARM64_T6000 Platform Release: 22.1.0

[NathanSiemers]

Yes, works for more than 1 inference step. Might need to update some of the huggingface docs if this isn’t considered a bug, because as I mentioned some docs recommend taking a single step to initialize the model before beginning the real image generation (a work-around for initialization problems with mps).

[patrickvonplaten]

Note that PNDM cannot work with just one inference step