refactor

We need a refactor. Maybe a redesign (breaking backwards compatibility). The codebase has accrued just a little too much tech debt. Particularly in the train_dalle.py and README.md

Training loop

• The training loop needs to be refactored into a function in the style of pytorch lightning. This dramatically improves readability and makes early returns possible which is a cleaner solution than raising an Exception.

Argument parsing

• All arg-parsing need to be well-consolidated into a single file. Perhaps another file at the root of the repo as it’s still something people are going to want to open up quickly.

Style guide and automatic formatting in CI

• This one’s easy - we need a consistent target format for multiple contributors to target. I like PEP8 but honestly if we just pick one I’ll be happy we’re using a linter/formatter at all. Alternatives are black/yapf. I could integrate this into Github Actions so that code is automatically formatted upon merge to main.

Configuration:

• we should provide reasonable defaults deepspeed_config.json in a folder at the root called “config” and encourage people to update that file rather than update the deepspeed_config variable in Python which is a bit of a hassle to get to and find each time. A comment at the top of each file with a links to:
• https://deepspeed.readthedocs.io/en/latest/schedulers.html # Various (otherwise) undocumented schedulers from DeepSpeed
• https://www.deepspeed.ai/docs/config-json/#zero-optimizations-for-fp16-training # Options specific to zero.

README.md

• The readme is a bit long; but that’s alright. I prefer the idea of a single source of truth with documentation; especially one that doesn’t require the internet after you’ve downloaded it e.g. the wiki
• It just needs a table of contents and a “quick start” at the top. I do like the idea of continuing to give progress updates on the current best at the top as well. So let’s keep that. taming-transformers has a wonderfully clean README.md with a table of contents which could be used as a basis.

Feel free to criticize the current design (please be constructive and polite - we’re all lazy coders at the end of the day) so we can decide if breaking backwards compatibility for a new design would be worthwhile.

[janEbert]

@afiaka87 with regard to the loop comment, I don’t disagree with putting the loop (or I’m guessing training step) into its own function. Just like I’d favor having a main with corresponding if __name__ == '__main__' guard. I was just referring to not including too many abstracting functions in it. 😃

Just to be clear, I agree with or am neutral towards what I didn’t write about!

[johnpaulbin]

Move this to Discussions 👍