Provide annotation that a test currently fails by given circumstances

Creating an issue as base for the PR for a @NotWorking annotation. Text comes from the PR by @Marcono1234

---

Often tests fail due to a bug in the tested application or in used dependencies. Traditionally such a test method would be annotated with JUnit’s @Disabled. However, this has the following disadvantages when the bug causing the test failure is fixed:

• the developer might not notice the existing test method and create a new one
• the existing test method might not be noticed and remains disabled for a long time after the bug has been fixed, adding no value for the project @NotWorking tries to solve these issues. Unlike @Disabled it still executes the annotated test method but aborts the test if a test failure or error occurs. However, if the test is executed successfully the @NotWorking annotation will cause a test failure because the test is working. This lets the developer know that they have fixed a bug (possibly by accident) and that they can now remove the @NotWorking annotation from the test method.

It is therefore recommended to prefer using @NotWorking over @Disabled in most cases; exceptions for this are for example ‘flaky’ tests.

Important: This annotation is not intended as a way to mark test methods which intentionally cause exceptions. Such test methods should use JUnit’s assertThrows or similar means to explicitly test for a specific exception class being thrown by a specific action.

[nipafx]

I still think that “expected”, especially in the context of unit tests, might erroneously give the impression that it is desired that the test fails, while it is actually merely “acknowledged”, see also my comment https://github.com/junit-pioneer/junit-pioneer/issues/551#issuecomment-1001058349 above.

I understand, but I still think it’s the best option.

What else (except for the renaming) has to be done for the pull request?

I’ll take another look and let you know over there.

Also, if you think it is easier, feel free to work on the pull request yourself (or use it as base for your changes); but please let me know in that case beforehand.

I don’t think I’ll do that, but good to know there’s the option (you never know when boredom presents itself 😉). Will let you know if that happens. 👍🏾

[nipafx]

To keep this issue and the related PR from stalling, I’m just gonna make the call and say the extension should be named @ExpectedToFail.

@Marcono1234 Are you ok with that? Or at least ok enough to finish the PR? 😉 If not, that’s ok, too, just let us know and we’ll finish it.