DreamBooth fails when training with 8bit adam optimizer.

Describe the bug

Seems that 8bit adam optimizer fails when training dreambooth.

The command I used:

python train_dreambooth.py
–pretrained_model_name_or_path=“CompVis/stable-diffusion-v1-4”
–instance_data_dir=$instance_folder
–class_data_dir=$class_folder
–output_dir=$model_checkpoints_folder
–with_prior_preservation --prior_loss_weight=1.0
–instance_prompt=“a photo of sks dog”
–class_prompt=“a photo of dog”
–resolution=512
–train_batch_size=1
–gradient_accumulation_steps=2
–gradient_checkpointing
–use_8bit_adam
–learning_rate=5e-6
–lr_scheduler=“constant”
–lr_warmup_steps=0
–max_train_steps=800

Reproduction

No response

Logs

RuntimeError: set_sizes_and_strides is not allowed on a Tensor created from .data or .detach().
If your intent is to change the metadata of a Tensor (such as sizes / strides / storage / storage_offset)
without autograd tracking the change, remove the .data / .detach() call and wrap the change in a `with torch.no_grad():` block.
For example, change:
    x.data.set_(y)
to:
    with torch.no_grad():
        x.set_(y)

System Info

• diffusers version: 0.4.0
• Platform: Linux-5.8.0-44-generic-x86_64-with-glibc2.29
• Python version: 3.8.10
• PyTorch version (GPU?): torch 1.10
• Huggingface_hub version: 0.10.0
• Transformers version: 4.22.2
• Using GPU in script?: <fill in>
• Using distributed or parallel set-up in script?: <fill in>

[hervenivon]

@paudom, I faced the same issue in a conda environment created with pytorch==1.10.2 and cudatoolkit=11.3.1. I solved my issue by running it inside a docker container using their latest aka nvcr.io/nvidia/pytorch:22.09-py3

[paudom]

@nanlliu Could you please tell me which Pytorch version or setup did you finally used? Thanks! I’m having the same issue.