add device map and accelerate to DiffusionPipeline abstraction to reduce memory footprint when loading model

Is your feature request related to a problem? Please describe. As a follow up for #281 we could add the device map and the possibility to load weights using accelerate to DiffusionPipeline abstraction for smaller memory footprint when loading models.

Describe the solution you’d like

from diffusers import DiffusionPipeline



pipeline = DiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-3", device_map="auto")

Describe alternatives you’ve considered

• Load models by hand every time
• Have a larger memory footprint when loading models

Additional context This is a follow up for #281.

I can work on that if you folks would let me.

[piEsposito]

@CrazyBoyM, what happens is that, for that feature to work, we need a version of accelerate with https://github.com/huggingface/accelerate/pull/747 merged. This PR was merged 4 days ago and the last release was 7 days ago.

Until they make the next release for accelerate, if you really want to use this feature I suggest you install accelerate from the master branch of the repository: pip install git+https://github.com/huggingface/accelerate.git, which you can revert to the pypi version after the next release.

[CrazyBoyM]

Hi, friend.I see this PR https://github.com/huggingface/diffusers/pull/361 but when i try this:

self.pipe = StableDiffusionPipeline.from_pretrained(
            self.model_id_or_path,
            revision="fp32", 
            device_map="auto",
            torch_dtype=torch.float32,
            scheduler = DDIMScheduler(
                beta_start=0.00085,
                beta_end=0.012,
                beta_schedule="scaled_linear",
                clip_sample=False,
                set_alpha_to_one=False,
            ),
            # use_auth_token=True,
        )
        self.pipe = self.pipe.to(self.device)

i get a error with:

   set_alpha_to_one=False,
  File "/root/.conda/envs/ai/lib/python3.7/site-packages/diffusers/pipeline_utils.py", line 517, in from_pretrained
    loaded_sub_model = load_method(os.path.join(cached_folder, name), **loading_kwargs)
  File "/root/.conda/envs/ai/lib/python3.7/site-packages/transformers/modeling_utils.py", line 2269, in from_pretrained
    max_memory=max_memory,
  File "/root/.conda/envs/ai/lib/python3.7/site-packages/accelerate/utils/modeling.py", line 480, in infer_auto_device_map
    max_layer_size, max_layer_names = get_max_layer_size(modules_to_treat, module_sizes, no_split_module_classes)
  File "/root/.conda/envs/ai/lib/python3.7/site-packages/accelerate/utils/modeling.py", line 261, in get_max_layer_size
    modules_children = list(module.named_children())
AttributeError: 'Parameter' object has no attribute 'named_children'

can you help me?