Truely random test order with posibility for externaly given seed

By default junit executes test in default order that is based on method name hash code. In java hashcodes of string doesn’t change between executions, so the order is random, but constant for given computer. That could lead to unpredictable errors.

Ref: https://github.com/junit-team/junit/blob/master/src/main/java/org/junit/internal/MethodSorter.java#L13

$ groovysh 
Groovy Shell (1.8.6, JVM: 1.7.0_79)
Type 'help' or '\h' for help.
---------------------------------------------------------------------------------------------------------------------------------------------------------
groovy:000> a = 'testSample'
===> testSample
groovy:000> a.hashCode()
===> 1710059740
groovy:000>
$ groovysh 
Groovy Shell (1.8.6, JVM: 1.7.0_79)
Type 'help' or '\h' for help.
---------------------------------------------------------------------------------------------------------------------------------------------------------
groovy:000> a = 'testSample'
===> testSample
groovy:000> a.hashCode()
===> 1710059740
groovy:000>

To fix this enchantment, there should be truly random order option to explicitly specify in test case using @FixMethodOrder.

I propose to follow Rspec --order random option implementation. It actually makes random order for execution but seed used for this is displayed and logged into reports. This gives developers a chance to execute the same execution with --order rand:3455. It guarantees that tests will be executed in the same order, therefore it’s possible to hunt for execution order bugs.

Cite:

When you use --order random, RSpec prints out the random number it used to seed the randomizer. When you think you’ve found an order-dependency bug, you can pass the seed along and the order will remain consistent:

–order rand:3455

Example execution:

$ rspec test.rb --order rand

Randomized with seed 64539

testing
  should equal 5
  should contain 'test'

Finished in 0.02 seconds (files took 0.0092 seconds to load)
2 examples, 0 failures

Randomized with seed 64539

$ rspec test.rb --order rand

Randomized with seed 12834

testing
  should contain 'test'
  should equal 5

Finished in 0.02 seconds (files took 0.0088 seconds to load)
2 examples, 0 failures

Randomized with seed 12834

$ rspec test.rb --order rand:12834

Randomized with seed 12834

testing
  should contain 'test'
  should equal 5

Finished in 0.02 seconds (files took 0.0083 seconds to load)
2 examples, 0 failures

Randomized with seed 12834

Rspec random refenrence: http://blog.davidchelimsky.net/blog/2012/01/04/rspec-28-is-released/

[cardil]

Hey @Endron. Your point is good but please look closer in Rspec --order random option implementation. It actually makes random order for execution but seed used for this is displayed and logged into reports. This gives developers a chance to execute the same execution with --order rand:3455. It guarantees that tests will be executed in the same order.

Cite:

When you use --order random, RSpec prints out the random number it used to seed the randomizer. When you think you’ve found an order-dependency bug, you can pass the seed along and the order will remain consistent:

--order rand:3455

This is an example execution (you can see the seed is displayed in command line): https://travis-ci.org/coi-gov-pl/puppet-jboss/jobs/81936397#L556

[Endron]

The problem here is that there are actually two test smells that are interconnected with each other:

1. Tests depending on the execution order
2. Non repeatable tests

Putting tests into a really random order would help with the first. As no test can predict what happened before he can not depend on any other test to have been executed before. By changing the order with each run you would also find tests accidentally depending on the execution order.

But if you change the execution order to random your tests might also become non repeatable. So if you run the tests on the same machine two times they may yield different results. This will most likely be because your tests are somehow coupled in some way. They should not be but they are. There is no easy way to deal with these results. What do you do if you try to deploy some software because everything was green all the time but than in the final test you by accident find the one execution order that brakes one test? Tests must ether pass or fail and they must do this repeatedly. (Or else someone will start ignoring failing tests and just rerun them to get them passing.)

This is most likely part of what let to the implementation as mentioned by @ffbit: It is a compromise to face both problems. By making the order unpredictable they made it hard for anyone to implement there tests to use it. By making it deterministic they ensured that test execution always comes to the same result.