Improve the precision of our integration tests

We currently have a rather low precision when testing our pipeline due to due reasons.

1. • Our reference is an image and not a numpy array. This means that when we created our reference image we lost float precision which is unnecessary
2. • We only test for .max() < 1e-2 . IMO we should test for .max() < 1e-4 with the numpy arrays. In my experiements across multiple devices I have not seen differences bigger than .max() < 1e-4 when using full precision.

IMO this could have also prevented: https://github.com/huggingface/diffusers/issues/902

[Lewington-pitsos]

@patrickvonplaten after some research I think I understand the issue.

The images are currently stored in a low-precision format (e.g. png) which prevents us from testing with any greater precision than 1e-2. Even if we convert the image into a numpy array this will not help since the image itself is missing precision.

What we need to do is store a numpy representation of the image, say somewhere inside https://huggingface.co/datasets which we can then download and use for comparison.

The way we could do this is by generating the test output, saving that output as a numpy image and uploading it.

This is what I plan to do tomorrow.

This stack overflow thread was very helpful in understanding the issue.

[Lewington-pitsos]

@patrickvonplaten I made a PR adding the files: https://huggingface.co/datasets/hf-internal-testing/diffusers-images/discussions/2