Inpainting example is using data from the masked parts with strength 1

Describe the bug

It is my understanding that if we provide strength=1 to the inpainting pipeline, that the model would try to fill the missing parts without taking into account what was in the image before (where the mask is).

It does not appear to be the case and is really apparent when trying to zoomout of a picture. In that case, the prediction will very often fill this area with white.

Using the following mask (open the image in a new tab to see the white border):

I create prediction first by keeping the white border in the init_image, then I create prediction by filling the white border with similar colors (with gimp).

keeping white border:

Filling white border with gimp:

Here is my result with the white border:

Here is my result with the filled border:

It is clear that the model is keeping data about the parts which should be cropped by the mask.

Reproduction

The code I used to generate the grids is here

Add 3 images above named zoomout_mask.png zoomout_white.png zoomout_filled.png in the /tmp/inpainting folder and run the above script.

Logs

No response

System Info

diffusers version: commit 5cbed8e0d157f65d3ddc2420dfd09f2df630e978

[nagolinc]

@leszekhanusz Adding the un-noised latents was what I originally tried, but @jackloomen pointed out here (https://github.com/huggingface/diffusers/pull/241#issuecomment-1226283542) that was causing the final inpainted image to be blurry.

Worth noting that @eteriaai has apparently figured out a much better method (https://twitter.com/EteriaAI/status/1563018093055553539), so hopefully we’ll find out what that is soon.

[nagolinc]

@leszekhanusz Interesting.

I added a hack here (https://github.com/huggingface/diffusers/compare/main...nagolinc:diffusers:inpainting-fix) that fixes this. But I wish I understood better why it was happening in the first place.

(this is using white borders and strength=1.0 and my changes)