test: separate integration tests to their own gradle task

Some tests are slow and flaky and not safe to run in parallel. We’ve added tags to these tests and would like to use them to define a separate test run.

We have TteTest and MteTest tags for these tests. The remainder of the tests (hopefully less slow and flaky) remain untagged. We will be treating TteTest and MteTest as equivalent for this new test task.

We need one gradle task (possibly the default test task?) that excludes tests with either of these tags,
and one gradle task that includes only tests with either of these tags.

This needs to be done for

• engine and/or engine-test projects in the main repo
• modules, which are configured by terasology-module under build-logic

engine-tests/build.gradle is probably the easiest place to start.

References

Terasology tests are written using JUnit 5 and run with the JUnit Platform.

• Tagging and Filtering—JUnit 5 User Guide
• Testing in Java Projects: Test Grouping—Gradle 6.8

Naming Questions

• Should we make new tasks for “tests with MTE” and “tests without MTE”, and keep test in its current behavior of running all JUnit tests? Or should we change the default test task to be the “without MTE” one?
• What should the “tests with MTE” task be named? Could be less specific than “MTE” to include other implementations in this category. integrationTest, simulationTest, flakyTest?

[jdrueckert]

Oh I was already thinking about the stages in Jenkins when I said we want to have (only) those two separate ones. I would expect gradlew test to still run all of them.

[jdrueckert]

@keturn Thank you for writing this up 👍 and @mahela97 thank you for volunteering to implement this 👍

I believe we don’t want to keep test in its current behavior of running all tests, but have two new tasks, one for the tagged tests, one for the non-tagged tests. I’d suggest going for the following naming:

• “unitTest” (or just “unit”?) - everything “MteTest” and “TteTest” tags
• “integrationTest” (or just “integration”?) - everything with “MteTest” and “TteTest” tags

IMO this makes sense because of the following expectations: Proper unittests are expected to be many, but small in scope (testing a small aspect of the code), fast-running and parallelizable. Integration tests (including end-to-end (E2E) and scenario tests as well as tests combining a few units) on the contrary are expected to be fewer, but bigger in scope, slower-running and likely not parallelizable.

While it’s possible that currently, our non-tagged (neither MTE nor TTE) tests are not pure unit tests but could also be categorized as integration tests, they are expected to be a lot quicker than MTE and TTE tests. If they turn out not to be, we can think about tagging them as well and grouping them with the MTE and TTE tests.