data-driven tests should execute in parallel

right now, the unit of parallelization is a feature file. this has been working fine for most teams but yep we can do better

mainly 3 things:

• each Scenario in a Feature should be a unit
• each example row in a Scenario Outline should be a unit
• each “loop” row in a data-driven call (where the argument is a JSON array) should be a unit

possible challenges: not sure of implications if there is a callonce involved - worst case it may be executed multiple times. we may have to duplicate the Background for each thread.

[nmccann]

@ptrthomas I just wanted to chime in that I appreciate the solution you’ve gone with which gives users the flexibility to keep the current behaviour if desired.

While scenarios shouldn’t depend on one another (ideally, although I’m in the same boat as @SreecharanShroff in that some scenarios have been made dependent on one another to minimize setup code) - there are other cases where deterministic ordering can be valuable. For instance, one of the deliverables on my project is a “SQL diff” which shows all SQL calls that have been introduced in a new version. We use our Karate tests (run with a thread count of 1) to exercise our REST APIs which in turn results in the SQL queries being output (with a custom logback configuration). But this relies on some consistency from one run to the next. Even still, it’s not fool proof but it’s sufficient for our purposes.

[ptrthomas]

0.9.0 released