Common API

How much of a common API do you expect there to be across different objects? For example, the unet.py file contains a Trainer, while other models seem to only rely on their forward method, but they have different APIs and signatures.

Same with pipelines, for example with the BDDM pipeline having the following arguments for it’s __call__ method: mel_spectrogram, generator, torch_device=None, while the DDIM pipeline has batch_size=1, generator=None, torch_device=None, eta=0.0, num_inference_steps=50.

Do we expect all of these models, pipelines, schedulers to have a common API at the end? Is that even possible with diffusers?

It seems like most arguments are similar, but with a few specificities for models, pipelines and schedulers. That’s where having a configuration system would arguably work quite well as it would show very visibly what each of them has in terms of arguments for customization.

This reminds me a bit of the do_lower_case problem we have in transformers: some tokenizers have it, some don’t, but users don’t necessarily understand that and try to use it for all tokenizers.

[patil-suraj]

Thanks @kashif . I think this is exactly what @anton-l proposed with task/modality specific sub-classes.

[kashif]

One thing that is sometimes missing with the models’s forward is the notion of a context or conditioning… typically for the image gen. there is no conditioning, however for other modalities conditioning might be needed so if we can also add that in the forward and by default make it none, it will be helpful!