[tests] where to put deepspeed + fairscale tests

As a split off from this comment https://github.com/huggingface/transformers/pull/10039#pullrequestreview-585482462 we need to find a new home for deepspeed + fairscale tests.

Currently there are under examples/seq2seq because they rely on finetune_trainer.py ( run_seq2seq.py once the transition is over).

@sgugger suggests to keep the seq2seq folder as simple as possible. We also have ds_config.json there that could be moved too.

Seeing what’s happening in the fairscale land - I think we will need a bunch of various tests there in the future too.

So where should we put the deepspeed + fairscale tests?

Ideally they should be put under main tests, since they are part of the trainer core, but I’m not sure whether reaching across the tests suite is a clean approach.

My fantasy is that one day transformers will have a few essential tools that aren’t examples, and those will then leave somewhere in the main tree, perhaps src/transformers/apps and then it’d be easy to have such tests under tests.

So suggestions for now:

1. create examples/deepspeed and examples/fairscale
2. create examples/distributed and perhaps have all those extensions tested in one folder
3. create a new 3rd test suite for integrations
4. create tests/deepspeed - but as voiced earlier I’m not sure how reaching across a test suite will work - need to try - also this proposes to change the current flat structure of tests.

Perhaps you have other ideas.

@sgugger, @patrickvonplaten, @LysandreJik, @patil-suraj

[stas00]

Thank you for the clarification, @sgugger. I will start working on that transition.

[sgugger]

Could you pelase clarify, do you prefer most/all of the examples/tests to be flat, or would grouping make things easier to make sense of - I’m asking since some tests come with extra files (as is the case with ds_config files) - so examples/tests/deepspeed, …

We can certainly have several files once they are all together in one folder. I’d just like the examples subfolders to be clean, our internal testing should be setup so it’s the easiest for us to understand/debug.