strange renderings, no idea how to better describe this issue

I found a funny bug, not sure what is happening here. see screenshots

Very weird partials and weird final result is a puzzle

Previous render on same machine (had higher resolution and higher amount of steps) I got the expected results:

The only thing I believe I changed from previous successful run was that I’ve dropped down resulution and number of steps so I can quickly proptotype and see what gives me best results.

Here are settings for the render that fails, hope you can confirm that you can replicate this. What do you think is happening here?

{
    "batch_name": "Ancient ruins in a Rainforest experiment",
    "text_prompts": {
        "0": ["Ancient ruins in a tropical jungle, matte painting by Esao Andrews, Trending on Artstation."]
    },
    "n_batches": 5,
    "steps": 100,
    "display_rate": 1,
    "width": 256,
    "height": 256,
    "set_seed": "random_seed",
    "image_prompts": {},
    "clip_guidance_scale": "auto",
    "tv_scale": 0,
    "range_scale": 150,
    "sat_scale": 0,
    "cutn_batches": 8,
    "cutn_batches_final": null,
    "init_image": null,
    "skip_steps_ratio": 0.33,
    "init_scale": 1000,
    "skip_steps": 0, 
    "perlin_init": false,
    "perlin_mode": "mixed",
    "skip_augs": false,
    "randomize_class": true,
    "clip_denoised": false,
    "clamp_grad": true,
    "clamp_max": "auto",
    "fuzzy_prompt": false,
    "rand_mag": 0.05,
    "eta": "auto",
    "diffusion_model": "512x512_diffusion_uncond_finetune_008100",
    "use_secondary_model": true,
    "sampling_mode": "ddim",
    "diffusion_steps": 1000,
    "ViTB32": true,
    "ViTB16": true,
    "ViTL14": true,
    "ViTL14_336": false,
    "RN101": false,
    "RN50": false,
    "RN50x4": false,
    "RN50x16": false,
    "RN50x64": false,
    "cut_overview": "[5]*400+[1]*600",
    "cut_innercut": "[1]*400+[5]*600",
    "cut_ic_pow": 1,
    "cut_ic_pow_final": null,
    "cut_icgray_p": "[0.2]*400+[0]*600",
    "smooth_schedules": false,
    "intermediate_saves": 25,
    "stop_early": 0,
    "fix_brightness_contrast": true,
    "high_contrast_threshold": 80,
    "high_contrast_adjust_amount": 0.85,
    "high_contrast_start": 20,
    "high_contrast_adjust": true,
    "low_contrast_threshold": 20,
    "low_contrast_adjust_amount": 2,
    "low_contrast_start": 20,
    "low_contrast_adjust": true,
    "high_brightness_threshold": 180,
    "high_brightness_adjust_amount": 0.85,
    "high_brightness_start": 0,
    "high_brightness_adjust": true,
    "low_brightness_threshold": 40,
    "low_brightness_adjust_amount": 1.15,
    "low_brightness_start": 0,
    "low_brightness_adjust": true,
    "gobig_orientation": "vertical",
    "gobig_scale": 2,
    "keep_unsharp": false,
    "symmetry_loss_v": false,
    "symmetry_loss_h": false,
    "symm_loss_scale":  2400,
    "symm_switch": 45,
    "interp_spline": "Linear",
    "max_frames": 10000,
    "sharpen_preset": "Off",
    "frames_scale": 1500,
    "frames_skip_steps": "60%",
    "animation_mode": "None",
    "key_frames": true,
    "angle": "0:(0)",
    "zoom": "0: (1), 10: (1.05)",
    "translation_x": "0: (0)",
    "translation_y": "0: (0)",
    "video_init_path": "/content/training.mp4",
    "extract_nth_frame": 2
}

I believe it is resolution, now setting to 500x500 render is back in business.

[kjhenner]

Also, @nikocraft, you might want to experiment with turning off the secondary model for smaller images to let the full diffusion model do its thing. At smaller image scales the diffusion model’s contribution to your time/ram characteristics is going to be minimal, so there’s probably not reason to scale it down.

[kjhenner]

I’m currently trying to do a bunch of runs at 512x512 with 100 steps to get representative samples from a bunch of artists. Getting good results across a bunch of artists at the small time/pixel scale is a bit of a puzzle. I’ll have to try @lowfuel 's suggestion of plms vs. vs ddim and perlin_init, though. I’m getting somewhat reasonable results for the most part, though definitely pretty grainy and low coherence for a lot of artists. If you want a reference, my current settings are as follows (with the text_prompt getting over-written):

    "batch_name": "samples",
    "text_prompts": {
        "0": [
            "A beautiful painting of a Castle in the Scottish Highlands, underexposed and overcast, by Asher Brown Durand, trending on ArtStation."
        ]
    },
    "n_batches": 4,
    "steps": 100,
    "display_rate": 10,
    "width": 512,
    "height": 512,
    "set_seed": 71,
    "image_prompts": {},
    "clip_guidance_scale": 200000,
    "tv_scale": 0,
    "range_scale": 150,
    "sat_scale": 0,
    "cutn_batches": 4,
    "cutn_batches_final": 2,
    "init_image": null,
    "skip_steps_ratio": 0.33,
    "init_scale": 1000,
    "skip_steps": 0, 
    "perlin_init": false,
    "perlin_mode": "mixed",
    "skip_augs": false,
    "randomize_class": true,
    "clip_denoised": false,
    "clamp_grad": true,
    "clamp_max": "auto",
    "fuzzy_prompt": false,
    "rand_mag": 0.05,
    "eta": "auto",
    "diffusion_model": "512x512_diffusion_uncond_finetune_008100",
    "use_secondary_model": true,
    "sampling_mode": "ddim",
    "ViTB32": 1.0,
    "ViTB16": 1.0,
    "ViTL14": false,
    "ViTL14_336": false,
    "RN101": false,
    "RN50": 1.0,
    "RN50x4": false,
    "RN50x16": 1.0,
    "RN50x64": false,
    "cut_overview": "[8]*400+[2]*600",
    "cut_innercut": "[1]*400+[8]*600",
    "cut_ic_pow": 0.8,
    "cut_ic_pow_final": null,
    "cut_icgray_p": "[0.2]*400+[0]*600",
    "smooth_schedules": false,
    "intermediate_saves": 0,
    "stop_early": 0,
    "fix_brightness_contrast": true,
    "adjustment_interval": 10,
    "high_contrast_threshold": 80,
    "high_contrast_adjust_amount": 0.45,
    "high_contrast_start": 20,
    "high_contrast_adjust": false,
    "low_contrast_threshold": 20,
    "low_contrast_adjust_amount": 1,
    "low_contrast_start": 20,
    "low_contrast_adjust": true,
    "high_brightness_threshold": 180,
    "high_brightness_adjust_amount": 0.45,
    "high_brightness_start": 0,
    "high_brightness_adjust": true,
    "low_brightness_threshold": 40,
    "low_brightness_adjust_amount": 0.85,
    "low_brightness_start": 0,
    "low_brightness_adjust": true,
    "gobig_orientation": "vertical",
    "gobig_scale": 2,
    "keep_unsharp": false,
    "symmetry_loss_v": false,
    "symmetry_loss_h": false,
    "symm_loss_scale":  2400,
    "symm_switch": 45,
    "interp_spline": "Linear",
    "max_frames": 10000,
    "sharpen_preset": "Off",
    "frames_scale": 1500,
    "frames_skip_steps": "60%",
    "animation_mode": "None",
    "key_frames": true,
    "angle": "0:(0)",
    "zoom": "0: (1), 10: (1.05)",
    "translation_x": "0: (0)",
    "translation_y": "0: (0)",
    "video_init_path": "/content/training.mp4",
    "extract_nth_frame": 2
}