Execution order of extensions changed with 1.7.0

Since 1.7.0 some extension annotations at class level are not applied before @RegisterExtension any more.

Specifically @SetSystemProperty is using beforeEach now but used beforeAll and has been executed before @RegisterExtension annotated extension fields in the test class.

We are using DropwizardAppExtension to bootstrap an application for all tests in the test class. In our case, the application needs the system property on startup to be configured for the test.

This change has been introduced with #496.

We see our test failing with the update to 1.7.0.

The PR and associated issues refer to alignment and consistent behavior. That sounds reasonable at a first glance. My personal assumption would be that something at class level is wrapped around something at method level and is applied as a class extension.

Do you have any suggestion how a test can be configured with 1.7.0 when it uses a static class extension that needs a system property to be configured for the test. In our specific case we could try to bootstrap the application in the test method and shut it down in a finally statement. But in general that is not suitable as it should be started only once for many test methods for performance reasons.

[beatngu13]

Hi @JoergSiebahn,

thanks for the feedback! 👍 So, we will first follow the aforementioned approach, where we “try to keep the current/new behavior, but make the extension additionally visible in @BeforeAll”.

PS: I only used the @Order annotation to make it more clear how the extension would behave. From the JUnit 5 user guide:

By default, test classes and methods will be ordered using an algorithm that is deterministic but intentionally nonobvious.

[beatngu13]

Due to our new reset mechanism, I think we can no longer combine BeforeEach/BeforeAll. But as @Michael1993 said, we could provide a configuration option:

@TestMethodOrder(OrderAnnotation.class)
// Default execution phase BeforeEach/AfterEach.
@SetSystemProperty(key = "foo", value = "bar")
class MyTestA {

	@BeforeAll
	static void setUpOnce() {
		assertThat(System.getProperty("foo")).isNull();
	}

	@BeforeEach
	void setUp() {
		assertThat(System.getProperty("foo")).isEqualTo("bar");
	}

	@Order(1)
	@Test
	void test1() {
		assertThat(System.getProperty("foo")).isEqualTo("bar");
		// Not visible in test2() because of AfterEach reset.
		System.setProperty("foo", "baz");
	}

	@Order(2)
	@Test
	void test2() {
		assertThat(System.getProperty("foo")).isEqualTo("bar");
	}

}

@TestMethodOrder(OrderAnnotation.class)
// Alternative execution phase BeforeAll/AfterAll.
@SetSystemProperty(key = "foo", value = "bar", executionPhase = BEFORE_ALL)
class MyTestB {

	@BeforeAll
	static void setUpOnce() {
		assertThat(System.getProperty("foo")).isEqualTo("bar");
	}

	@BeforeEach
	void setUp() {
		// Before test1() "foo" = "bar", after test() "foo" = "baz".
		assertThat(System.getProperty("foo")).isNotNull();
	}

	@Order(1)
	@Test
	void test1() {
		assertThat(System.getProperty("foo")).isEqualTo("bar");
		// Visible in test2() because of AfterAll reset.
		System.setProperty("foo", "baz");
	}

	@Order(2)
	@Test
	void test2() {
		assertThat(System.getProperty("foo")).isEqualTo("baz");
	}

}

Thoughts?