[feature] Enable overriding of `config.tests` for `run-multiple`

Hello, first thank you for the work you have put in here!
(This is the second part of a feature request. For the first part please go to #976.)

Sidenote We at @trivago Leipzig use CodeceptJS for end-to-end testing and are working currently on a simple way to run multiple jobs in parallel without using grep and tags.

Feature A first step into having a working solution is to enable CodeceptJS to load tests not only by grep/tags but also by their concrete file name. As this is already working for the run command it would be a useful enhancement to CodeceptJS itself when the user is able to create multiple groups and override inside them the tests that should be run.

The true power with this feature lies in combination with #976. If both features get accepted it will be possible do dynamically create run-multiple configuration by the amount of overall scenarios and distribute them evenly between a defined amount of runs. For a working examples please have a look here and here.

Technical explaination I’d like to propose to extend the runSuite function to also pass the tests configuration down from the multiple configuration to the CodeceptJS runcall.

Example

{
  "tests": "./*_test.js",
  "multiple": {
    "default": {
      "browsers": ["chrome", { "browser": "firefox"}]
    },
    "group1": {
      "tests": "./*_group1_test.js",
      "browsers": ["chrome"]
    },
    "group2": {
      "tests": "./*_group2_test.js",
      "browsers": ["chrome"]
    }
  }
}

Todos

• Add test-runner service in docker-compose.yml
• Write test case and verify nothing is breaking
• Adapt runSuite function
• Create a Pull Request in the fork https://github.com/sveneisenschmidt/CodeceptJS/pull/2
• Feedback by maintainers
• Create a Pull Request in this repository

What do you think?

PS: If you want to see why glob support is needed please look at my demo repository:

• https://github.com/sveneisenschmidt/codeceptjs-boilerplate#todo
• https://github.com/sveneisenschmidt/codeceptjs-boilerplate/blob/master/lib/create-multiple.js
• https://github.com/sveneisenschmidt/codeceptjs-boilerplate/blob/master/codecept.conf.js

[sveneisenschmidt]

@DavertMik @reubenmiller Thank you for the great feedback! I filed the PRs but got violations by Hound. Going to fix them soon.

I will join Slack and then we can further discuss how to implement dynamic multiple runs in a simple way. I have already an idea, but it relies on these concrete features.

Thank you!

Update: Added my first ideas https://gist.github.com/sveneisenschmidt/02c98a4a416ec8f3610ec6409e16393e

[DavertMik]

Yes, this looks like a good start to a long road. So that’s why I’d propose to add @sveneisenschmidt to our dev channel in Slack and continue there.