JUnit needs a better story for extensibility

This issue is a continuation of the discussion in #1158. This is the essential issue, and it prevents me from using JUnit for anything other than toys:

It’s starting to sound like the abstractions that JUnit provides are inadequate.

On the one hand, we’ve got Runner, which is potentially very powerful, but that power can only be accessed through subclassing. I can’t compose two different Runners to combine their effects.

On the other hand, we have TestRule, which is interface-based and composes well, but which isn’t powerful enough to safely extend or modify the way test methods are executed, therefore making them largely useless for anything other than simple callbacks (e.g. TemporaryFolder, ErrorCollector, Stopwatch) and filters (Verifier). The main exception is Timeout.

There are tons of interesting and useful custom test runners for JUnit, all of which are utterly non-interoperable with each other. They just don’t compose, and there are really fundamental reasons why they can’t compose under the current architecture. JUnit 5 would be a fantastic opportunity to tackle this problem. Is this something that people have been thinking about for a while? Is there a wiki or something that discusses potential design directions? The only things I’m aware of targeting JUnit 5 are (1) removal of Hamcrest matchers (yay), (2) removal of a really old runner (yay), (3) targeting of JDK6 (what’s the point? Do you need ArrayDeque or something?).

[kcooney]

These should be addressed in JUnit 5

[jlink]

Those topics (extensibility and runtime model) are also part of the “JUnit 5/Lambda crowdfunding campaign” which will start later this month or early in August. We’re in the final stages of setting up the campaign, and development (if the funding goal is reached) will probably start in October.

Of course, any additional prototyping and concrete experimentation is welcomed. What I don’t think makes a lot of sense is just collecting requirements and wishes since we already have many more than could be reasonably covered in a few months of work.

cheers, Johannes

2015-07-13 7:00 GMT+02:00 Kevin Cooney [e-mail]:

@rschmitt https://github.com/rschmitt all that I was trying to say is that if someome creates a new model for running tests, and tests using that model cannot be run from the existing APIs in JUnitCore and Request, then someone will either need to work with the plugin owners to use the new APIs, or they will be limiting their user base.

Assuming we want to support incremental discovery of tests, perhaps a dynamically created test would need to be associated with a parent test case (not test suite). When you run new-style tests using the JUnit 4 APIs, failures in the dynamically created test would me associated with the parent.

But I think we are getting ahead ourselves. We need a few people passionate about this idea to create a new project (probably on github) that depends on JUnit, and start experimenting with ideas. If things look promising, then we can look into compatibility with “legacy” JUnit tests. I personally agree with you that requiring forward and backward compatibility is too big of a constraint during the early prototyping phase, but when to worry about which challenges is entirely up to this new team.

Do you think you have time to work on this? Unfortunately, I won’t have time for the foreseeable future.

— Reply to this email directly or view it on GitHub https://github.com/junit-team/junit/issues/1161#issuecomment-120813293.