Discrepancy in scheduler `offset` usage

What API design would you like to have changed or added to the library? Why?

@jonatanklosko brought up an issue with some confusing code (why are alphas indexed differently across PNDMScheduler and DDIMScheduler), and some inconsistency in the offset usage.

Two questions:

1. Why does only one shift by offset?
2. Can offset be changed from an arbitrary int to a bool setting that shifts by 1 (the integer shift is more confusing if the only use case is stable diffusion inference with offset = 1)

Discussion began in #444, see here

https://github.com/huggingface/diffusers/blob/8eaaa546d89f836b716e92348786d878f883ee86/src/diffusers/schedulers/scheduling_pndm.py#L338-L339

https://github.com/huggingface/diffusers/blob/8eaaa546d89f836b716e92348786d878f883ee86/src/diffusers/schedulers/scheduling_ddim.py#L204-L205

I can tell you that the PNDMScheduler and DDIMScheduler have indexed the alpha’s differently for a while. Before introducing offset, PNDMScheduler had

alpha_prod_t = self.alphas_cumprod[timestep + 1]
alpha_prod_t_prev = self.alphas_cumprod[timestep_prev + 1]

And DDIMScheduler had

alpha_prod_t = self.alphas_cumprod[timestep]
alpha_prod_t_prev = self.alphas_cumprod[timestep_prev]

At minimum, we can figure this out and document better!

[jonatanklosko]

By timesteps[n] you probably meant alphas_cumprod[n] in the list of cases above, no?

Also here:

alpha_prod_t_prev = self.alphas_cumprod[prev_timestep + self._offset] if prev_timestep >= 0 else self.final_alpha_cumprod

I don’t see why we need to add the offset. For example, at timestamp 101, prev timestamp would be 1, so we want to take alphas_cumprod[1] rather than alphas_cumprod[1 + self._offset].

[patrickvonplaten]

Just to summarize, I think by having both:

steps_offset and set_alpha_to_one in the config as you suggested @jonatanklosko we should enable all use cases that we need. We should then indeed document this nicely.

@jonatanklosko very cool summary here, let me know if you need any help 😃