Extensions / Annotations to remind user about unfinished work

In the past few weeks several issues and PRs came up with ideas for extensions/annotations to “remind” the user a certain test

• is disabled because it fails until a fix / the propper implementation
• works now, but the underlying implementation has mistakes or needs to be refactored / has technical debts)

The following came up:

• Annotation that at test fails from a certain date on (@FailsAt #549) to remind your self to finally fixing/refactoring the underlying code
• Disable a test until a certain date (#366) or until a certain issue has been solved (#367)
• Annotation to mark a test (and inverting the test result), that the current test fails, because the test is correct, but the implementation has bugs. (@NotWorking #546)

So we came to a point where we write tests cheer, but they do not work (yet) cry. Joke aside all those written above seems to be annotatios/extensios to give the user the possibility to place a reminder at test level which automatically warns you at some point that there is work to be done (cause the test then fails).

At our team meeting this week we decided to create an (this) issue to have global look at these ideas and

• see if we find more use cases which fit into the scenario of “theres work to be done, but we don’t / can’t do it now - so please JUnit remind me about that”
• review the names of those any maybe find a communal annotatio name to show that all these belong to the same kind of topic

Distinction:

We also have issues / extension to

• disable tests, when other tests fail (#467 / #505)
• disable tests, if they have a certain name (#163)

But I don’t see those in the same category, because their goal is to reduce the number of executed / failing tests because unfulfilled dependencies / requirements.

[Michael1993]

I just had a talk with @verhas about this and he brought up an interesting point: @FailAt and @DisabledUntil are incorrect from an architectural standpoint.

Unit tests are supposed to be independent from external factors - including time.

Let’s say you have a library, where you have a couple classes and some tests. Some (one or more) tests are annotated with @DisabledUntil or @FailAt. You create a release. A release is supposed to be stable, i.e.: I can download and build it myself (If I recall correctly maven artifacts have source jars that include all the source code, including tests.). However, because some of your tests have what is essentially an expiration date, suddenly your artifact is not stable - if I download it and try to build it after the expiration date (set in an annotation), the build will fail.

We should include a warning about this in the documentation (at the very least). Alternatively, if we could make it so that these annotations behave differently in a release (how would we do that… no idea) that would be even better.

[helpermethod]

Maybe something like

@HasToBeFixedAfter -> @DisabledUntil
@HasToBeFixedBefore -> @FailAt
@HasToBeFixed -> @NotWorking