Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Question about CLIP-guided SD

See original GitHub issue

In cond_fn of examples/community/clip_guided_stable_diffusion.py, we have

# "predicted x_0" of formula (12) from https://arxiv.org/pdf/2010.02502.pdf
pred_original_sample = (latents - beta_prod_t ** (0.5) * noise_pred) / alpha_prod_t ** (0.5)

fac = torch.sqrt(beta_prod_t)
sample = pred_original_sample * (fac) + latents * (1 - fac)

I am confused that why not use pred_original_sample to get the image for CLIP guidance. Why do we need this extra conversion to get sample?

Issue Analytics

State:
Created a year ago
Comments:10 (4 by maintainers)

Top GitHub Comments

1reaction

patil-surajcommented, Oct 26, 2022

Hi, I adapted this example from the clip guided GLIDE example by @crowsonkb https://github.com/crowsonkb/glide-text2im/blob/main/glide_text2im/gaussian_diffusion.py#L293, I don’t have good intuition about this, maybe @crowsonkb could share more 😃

0reactions

yujianllcommented, Dec 10, 2022

I see, thanks!

Top Results From Across the Web

CLIP Guided Stable Diffusion (outdated, new guide coming ...

CLIP Guided Stable Diffusion (outdated, new guide coming soon). EDIT: I've overhauled the entire codebase! It's different now! but everything should be ...

CLIP-Guided-Diffusion - a Hugging Face Space by EleutherAI

CLIP Guided Diffusion HQ · skip_timesteps · clip guidance scale (Controls how much the image should look like the prompt) · tv_scale (Controls...

"CLIP Guided Diffusion 512x512, Secondary Model Method ...

question: do you know or wanna guess how to modify this latest stuff to be able to do resolutions that aren't square? i...

Clip-guided local duodenectomy for safe and minimal ... - NCBI

Clip-guided local duodenectomy is a safe and useful surgical option for minimally local resection of nonampullary duodenal neoplasms such as ...

Generates images from text prompts with CLIP guided diffusion.

It uses a 512x512 unconditional ImageNet diffusion model fine-tuned from OpenAI's 512x512 class-conditional ImageNet diffusion model (https://github.com/openai/ ...