Images are black when only using GPU (CUDA 11.7) along with NSFW warning

Describe the bug

I am new to AI in general, and I am trying to use the diffusers package by following the hugging face tutorial for running the model online and locally. But both of the methods (GPU) gives me a black image in return. However when I use the CPU mode pipe.to("cpu"), I get the results as expected.

In this case I am using the CompVis/stable-diffusion-v1-4 model that I got from using the following snippet:

git lfs install
git clone https://huggingface.co/CompVis/stable-diffusion-v1-4

Reproduction

from diffusers import StableDiffusionPipeline
import torch


pipe = StableDiffusionPipeline.from_pretrained(
    "D:\\4_Projects\\9_AI\\stable-diffusion-v1-4",
    revision="fp16", 
    torch_dtype=torch.float16,
)

pipe = pipe.to("cuda")

prompt = "a photo of an astronaut riding a horse on mars"
pipe.enable_attention_slicing()
image = pipe(prompt).images[0]

image.save(f"{prompt}.png")

The fixes I tried:

1. Editing the safety_checker.py

Editing the line has_nsfw_concepts = [len(res["bad_concepts"]) > 0 for res in result] to has_nsfw_concepts = [False for res in result]

1. Using a different seed (1024, 3214)
2. Using a different scheduler (LMSDiscreteScheduler)

Logs

The following is the only things I got in the terminal using the local model.

100%|██████████████████████████████████████████████████████████████████████████| 51/51 [04:32<00:00,  5.33s/it] 
Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed.
PS D:\4_Projects\9_AI\text_to_image_python>

System Info

• diffusers version: 0.9.0
• Platform: Windows-10-10.0.19044-SP0
• Python version: 3.10.8
• PyTorch version (GPU?): 1.13.0 (True)
• Huggingface_hub version: 0.11.1
• Transformers version: 4.25.1
• Using GPU in script?: yes
• Using distributed or parallel set-up in script?: I do not understand what distributed or parallel setup could be in my script. However I’ve read an article where accelerate is used for distributed setup, and I do have accelerate package installed.

[pcuenca]

I was going to suggest the same as @averad. @1MochaChan1, please let us know if that works 😃

[palant]

I looked through the memory optimizations on https://huggingface.co/docs/diffusers/optimization/fp16 and found that torch.backends.cudnn.benchmark = True fixed this, so now I can run on the GPU. The alternative is using a bunch more tweaks to bring memory usage further down and run with float32 which is considerably faster anyway for some reason.