Global JUnit rules revisited

A word in advance: I never have written such an extensive request in such a major project before, so please bear with me. 😃 My goal was to provide a comprehensive background as well as concrete suggestions for a solution. I hope that does not seem impolite 😃

Due to the topic dating back as far as 2009, I tried to sum up previous discussions, so this issue has become a bit lengthy (sorry 😕).

TLDR:

• some users in the JUnit community would love to have global rules
• I coin these global rules Guards, to differentiate from normal TestRules, RunListeners and Runners which mostly are scoped to test class level only
• Guards should be deactivated by default, with a global opt-in so they don’t have to be declared inside every test class.
• Guards should not cause too much friction with existing code or tools (e.g. Maven Surefire)
• I would like to contribute time and code to solve the problem (as a prototype at first, then as a real solution)
• Your feedback is very much appreciated 😃

Problem setting

At our company we want to run a set of global checks against each test to ensure it complies with company standards. If not, the according tests should fail.

Some actual use cases are:

• nothing is printed to standard error
• our dependency injection container has been shut down properly in @After(Class)#tearDown()).

There are more cases I can name on demand, but I wanted to keep the list short. All these cases have in common that they are some sort of global TestRule. In some cases it’s not that obvious that a rule has been violated or someone simply forgot a line of code(nobody is perfect).

History

“Global rules” have been initially requested in 2009 on the JUnit mailing list, which resulted in #69 Global rules. This discussion then has been continued in #444 JUnit should have an exhaustive listener framework in 2012. I will pick up some suggestions from these issues far further below. Besides the discussion of other use cases, @BenRomberg has been making some important arguments for having such a feature.

Some of the main arguments against global rules have been

• integration with other tools that use their custom runners, like Maven or other IDEs
• Tests should not fail due to some “obscure” magic (package level annotations, junit.properties).
• The same could be achieved with: TestListener, Runner, …

However it seems a part of the community is asking for a long time. There has been a lot of discussion and I guess that JUnit also evolved much more in the past 3 years (hence “revisited”).

Tried approaches

We tried to solve the “no standard error output” and the “container is always destroyed problem”. These are basically additional assertions/rules after the tests in a test class are run.

• For the first, we tried using a JUnit RunListener configured into Maven Surefire. However the RunListener is removed from the test run if an exception (such an AssertionError thrown by Assert.* methods) is thrown. My colleague @globalworming asked on Stackoverflow (suggesting using a custom Runner) and the JUnit mailing list, however there was no global solution. As stated by @dsaff in #444 RunListeners are read-only and I totally agree this should stay the way it is.
• Using a custom runner would not work with Parameterized tests(which also requires @RunWith(Parameterized.class)) and adding a rule (a set of rules) to every single class is not an option.
• Using a “company TestRule” which delegates to all other rules we need would still require changing every test class.

Suggestion

With hundreds of test classes in mind, changing every single one of them is no viable option.

To distinguish from existing terms (such as runner, listener or rule) which are declared in classes, I would like to introduce the term Guard. Here, imagine a historic city with solid high walls and one or more guards standing in front. These are checking on people (tests) going in(@Before) and out(@After). Any people which fail their criteria will be put to jail(AssertionError).

By default there are no guards in front of the city and everyone is allowed in and out as they please (current JUnit), although there might be laws in the city (->TestRule) for certain groups of people (test classes). You can then opt-in to place Guards in front of your city gates.

Some folk tend to come to the city and, as they leave, they are shouting mean things. This is a very polite city, so we tell the guards to put these visitors into jail for their offensive behaviour. That is, we ensure there is no output on standard error. (I can also provide a more comprehensive example, but I left it out to keep things short)

My intention using this analogy was to have an abstract, extensible story for this problem. This takes the focus away from the details towards the main problem.

How to implement this?

I’m not an JUnit expert as many of you are, so I would be glad for ideas and feedback especially on this part.

Before opening this issue, I already tried taking a look at JUnit’s and also Maven Surefire’s source code to evaluate some options. The goal is to create a global opt-in based approach to Guards which doesn’t interfere with existing code, such as test runners + annotations (e.g. parameterized tests). The solution should be resilient to human failure, i.e. it should be possible to declare it in one place instead of for every test class.

RunListeners can already be declared in the JUnitCore. As far as I know, this is JUnits common entry point for all applications, such as IDE’s, build systems, command line, etc. Adding guards at this point would allow seamless integration with existing test runners (e.g. @RunWith(Parameterized.class)) and leaving existing test code unchanged.

Confguring Guards

A common question in the past was how Guards should be configured. I could think of the following options

1. Adding a method addGuard(Guard) to JUnitCore. I think this should always be possible, as external applications (e.g. an IDE, Command line tools or Maven Surefire) can integrate themself here (i.e. adding a Guard to the Maven build via Surefire’s configuration)
2. Having a junit.properties file. However I fear that it takes away much of the flexibility(extensibility) available in normal source code. Also this is error prone to renaming tests/packages and simply not inside the code, which is the way users are used to.
3. Automatic discovery of configured guards using (package) annotations and normal source code. Renaming things inside an IDE would also integrate with the according changes in the configuration automatically. Also exclusions can be managed on source code vs properties file level.

(Note: The basic ideas of 2. and 3. were already mentioned in #69 and #444) Any of these options would allow leaving pre-Guard code in user projects as is (no change required whatsoever).

From my point of view options 1) and 3) are the best, with 3) extending 1) with auto-config.

To keep the initial post shorter, I will put my suggestion for a package-based configuration into a separate comment below.

Inside a Guard, it would be nice to reuse the normal JUnit infrastructure using rules and class rules:

public class MyGuard {
  // Checks before/after each test class
  @ClassRule
  public final NoContainerExistsRule noContainerExistsRule = new ContainerExistsRule();

  // Checks before/after each method
  @Rule
  public final NoSystemErrorRule noSystemErrorRule = new NoSystemErrorRule();

  // Chaining rules would also be possible if the @Rule annotations above are removed
  @Rule
  public final RuleChain dependantRule =
    // This would check that destroying the container does not output to stderr
    RuleChain
    .outerRule(noSystemErrorRule)
    .around(noContainerExistsRule);
}

This comes quite close to the TestFragments idea in #444. (Side note: Guards must not contain test cases 😇 )

What do you think of my ideas so far? Would it be good if MyGuard implemented a certain interface (similar to RunListener, but with the ability to fail tests) for lifecycle events? I’m not certain, but if we used @Before and @After annotations any JUnit user could easily create/adapt his/her own Guards.

On the technical side, it may be better to manage Guards similar to the existing RunListener/RunNotifier code? However, because to the “Exceptions remove the listener” rule, we must not reuse this.

If a guard fails, the error should be reported in a way that it’s very clear where it came from, e.g.

AssertionError: A rule in com.example.MyGuard declared in the package-info.java file for package com.example failed.
  at org.junit.runner.JUnitCore.run(Runner):132
  [...]
Cause: AssertionError: There has been output to System.err
  at NoSystemErrorRule:42

This should weaken the “my test fails and I don’t know why as I didn’t declare anything in the test class” argument a bit.

Possible next steps

I really would like to contribute to JUnit if possible, so my proposal would be the following:

• to discuss possible approaches and find a consensus on open questions
• after that, I’d like to implement a prototype according to JUnit’s contribution guidelines
• Make the prototype production-ready and ship it as part of the next upcoming release as soon as possible to receive more feedback from the community. For that reason, I would like use JUnit 4.x as a foundation (porting it to JUnit 5 afterwards).

So, what do you think? 😃

[m9aertner]

Using a “Global Rules” approach, I was able to replace an even lower-level solution using AspectJ that had the goal of fixing certain things centrally, i.e. without having to change the unit test classes. While this is not a shippable amendment right now, I’d like to share some of the points that I found along the way:

• We have thousands of unit test classes. With and without base classes, JUnit 4 mostly, some JUnit 3. Points mentioned above like “just add base classes”, “ask the developers to…” and “use @RunWith on the unit test class” were not so attractive. Global rules are great because they allow one to change lots of things centrally which is great at a certain scale.
• We’re on JUnit 4.12, by the way, and call tests from Gradle via Ant and IDEs.
• My entry point is AllDefaultPossibilitiesBuilder. This class determines which “Runner” to use for a specific test. Unfortunately, I did not find a way in JUnit to inject a variant of this class using JUnit configuration, so I leveraged an existing classpath ordering mechanism to inject my variant implementation.
• For the JUnit 4 and JUnit3 cases, I create new ClassRunners which extend the original implementations, i.e. BlockJUnit4ClassRunner for JUnit4.
• Extending these is not super straightforward, as some methods are private (withClassRules is, withAfterClasses is not, this is inconsistent), but doable. It’s a different discussion on how to improve the extensibility of JUnit …
• That Runner adds further “Statements” to the existing sequence. This is done on two levels: class rules and method rules.
• I read the rules from a GuardingRules Java file, converted to TestClass, which is on the side and not directly connect the the unit tests being guarded. Code is available in JUnit for reading the rules off that TestClass and for running the rules in a custom Statement (RunRules).
• For now, I’ve hard-coded that GuardingRules.java file. I’ve toyed with above-mentioned approach to annotate package-info.java classes, but that did not work satisfactorily. I though it would be nice to find the “nearest” annotated package-info, walking to the root. That worked, but it looks like one cannot have a top-level (default package) package-info file. This means for each of our top-level packages (com, org, …) we’d have to have (likely redundant) guards files. Not good for us, but smaller codebases probably have a single root anyway, so this may not be an issue.
• For JUnit3 it was possible to wrap the existing runner inside the above Rule-based one, so we’re mixing rules and JUnit 3 for these cases (so far this looks good, but clearly that’s most experimental).
• And finally: the classpath injection approach works nicely in all environments, i.e. Eclipse, IDEA, and on the command line, without any re-configuration of the environment.

And why all this? This appraoch allows us to do monitoring (limit execution time, i.e. global @Timeout), logging and resource control globally. Also this is granular on @BeforeClass, test method(s) and @AfterClass level! (@Before/After is lumped together with the method execution, though. Would need more code to separate these apart).

Cheers, Matthias

[Unlimity]

@m9aertner You’re welcome.