Support configuring virtual suite created when running multiple suites with `__init__.robot`

I’m fairly familiar with Testcase and Suite Setups/Teardowns and use them heavily in my projects. But since I have multiple suites (which each represents a different hardware system with its own suite setup and teardown), each containing up to 60 testcases (also with setup and teardown), Robot Framework misses the option of a final “Run” teardown.

Some systems are not tested all the times, so these suites are deselected for specific runs (handled by RED GUI “Run Configurations”). Currently I implemented a “final” independent suite, which has some clean-up and logging/informing mechanisms implemented. Drawback of this implementation is, that this also counts as “passed testcase” and, the more problematic point: it will not get executed if a fatal error occurs.

Handling those clean-up mechanics is also not possible in one of those suite teardowns, since it is unclear, which suite will be executed last or even executed at all. And since all teardowns are somewhat generic and differ only in one or two parameters, it is unfeasible to have a special teardown for last suite.

My suggestion would be (if not already done, I could not find any implementation or documentation) to have an additional command line argument (--overall-teardown or --general-teardown) which points to a suite-file, where all the necessary final tear-down cases are written down. Then, it also has to be clarified in what way this will interact with rebot or if it could even be included in rebot.

Is this a suitable suggestion for RF?

Additional Info:

• Robotframework 3.2.2
• RED - Robot Editor 0.9.5
• Python 3.7.9 x64
• Windows 10 x64

[pekkaklarck]

Typically you accomplish this by having a nested test suite directory with a top level directory having an __init__.robot file with whatever suite teardown you need. If I got it right, you are running tests like robot tests.robot more.robot and thus don’t have such a natural place for a teardown. Notice that when running tests like this, Robot automatically creates a new “virtual” suite containing the executed suites as its child suites.

I understand the need for a top level teardown when executing multiple suites, but having a separate command line option to specify the teardown has some problems:

• If you just specify that you want to run My Keyword as a teardown, Robot doesn’t know where it finds the keyword unless you tell it somehow. In __init__.robot files this isn’t a problem because you can use normal library and resource imports. This could be handled, for exapmle, by requiring using long name like mylib.My Keyword and Robot then automatically importing mylib as library, but that’s not too convenient.
• We’d also need similar setting for specifying the top level setup.

An easier and also more flexible solution would be making it possible to specify an __init__.robot file to use with the virtual top-level suite when running multiple suites. We could make it so that if one of the executed files is __init__.robot, then that’s used to configure the virtual suite exactly like __init__.robot files typically configure suites created from directories. Basically the usage would be like robot __init__.robot tests.robot more.robot. A nice benefit of this approach is that it ought to be pretty easy to implement.

Would this work for you @far3ll? Would someone be interested to look at implementing this? As I wrote, it shouldn’t be too complicated but I won’t have time for it in the near future myself. Would be happy to review PRs and help otherwise if needed.

[pekkaklarck]

Yeah, executing a directory and having __init__.robot files on different levels makes it easy to have whatever granularity with setups and teardowns you need. I also agree sending emails etc. should typically be handled outside RF tests.

Anyway, sometimes running individual files or directories is convenient and being able to configure the virtual suite created in these cases would be handy. Doing that with __init__.robot files would be convenient, consistent and also pretty easy to implement.