[Community Pipeline] Modified Cross-Attention - Structured Diffusion Guidance for Compositional T2I synthesis

Intro

Community Pipelines are introduced in diffusers==0.4.0 with the idea of allowing the community to quickly add, integrate, and share their custom pipelines on top of diffusers.

You can find a guide about Community Pipelines here. You can also find all the community examples under examples/community/. If you have questions about the Community Pipelines feature, please head to the parent issue.

Idea: Modified cross-attention mechanism

This pipeline aims to implement this paper to Stable Diffusion, improving interpretability of the prompts. Some results of the paper Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis:

[20RitikSingh]

I want to work on this.

[keturn]

This is a super interesting paper! Note that there is a reference implementation included in the .zip of “supplementary material” on the paper submission. Also note that implementation is not released under the Apache License, so I don’t know if 🤗 can accept a PR that includes it. It might be safer to do a clean-room implementation from only the description in the text of the paper.

[I Am Not A Lawyer and I Am Not Your Lawyer, but the paper’s authors are obligated to remain anonymous until the end of the ICLR 2023 review period, so they might have a hard time speaking up for themselves right now.]

If you do choose to use the code from the supplementary material, you may need to swap some things around to make it better fit diffusers instead of ldm.