CI / Tests for DALLE-pytorch

As the number of papers (and pull requests) is increasing, the idea of testing a branch before merging it to master becomes more and more important.

@afiaka87 ans @rom1504 had the idea to include DALLE-Pytorch into a continuous integration cycle or at least run a few tests before merging a branch to ensure no essential features get broken.

A simple, first test could look as follows:

run the follwing commands (each with text-image-folders and a wds stream, as well as with open-AI and taming transformer):

• python…
• deepspeed…
• deepspeed fp16
• deepspeed apex

The tests can be speed up by

itertools.islice(dataloader, 10):`“ if Test=True else dataloader

So the tests only run for a few samples.

[rom1504]

Why can’t the most important tests be run on CPU ? Even if it’s takes 5min to do 3 steps of batch size 2 on cpu, it’s good enough

[afiaka87]

This is really cool! Also the colab (see the WDS implementation, I tested it on your colab #280 (comment)).

But I think it might get a little flickery if we test only using colab - a few simple automatic test steps could still be useful where you do not have to rerun the colab every few seconds and change the parameters manually.😃

Do you have anything in mind other than colab which can be spun up that easily? @rom1504 mentioned running like 10 steps on the CPU; but if that winds up being “a whole thing” then I probably won’t spend too much time on it. Have honestly never tried with DALLE (or anything really) - but I’m guessing it doesn’t work as it stands?