Android Test Orchestrator support

Hi, I’ve tried running tests with Android Test Orchestrator, however it seems to be broken now (it works without Orchestrator though). My code: https://github.com/zawadz88/ExoPlayerAudioSample/commit/61257d3fc9f8edd57a8d40eed4ee7ccbc7b8fa7d

I’ve added two tests in a single class like this:

package com.github.zawadz88.exoplayeraudiosample.test
// imports
class MainActivityTest {

    @Suppress("unused")
    @JvmField
    @RegisterExtension
    val scenarioExtension = ActivityScenarioExtension.launch<MainActivity>()

    @Test
    fun play_button_should_be_displayed() {
        // some test
    }

    @Test
    fun prev_and_next_buttons_should_be_disabled() {
        // some other test
    }
}

When I setup Android Test Orchestrator and run the tests, AndroidJUnitRunner finds the tests on first run but creates an invalid Test description which makes it impossible to run them.

An expected test execution here would be:

1. We run the tests from command-line/Android Studio
2. On first test execution AndroidJUnitRunner collects a test list. In my case this should add two tests - com.github.zawadz88.exoplayeraudiosample.test.MainActivityTest#play_button_should_be_displayed and com.github.zawadz88.exoplayeraudiosample.test.MainActivityTest#prev_and_next_buttons_should_be_disabled
3. Actual tests should be passed as test arguments to AndroidJUnitRunner under a class key.

What actually happens is that in 2. MainActivityTest#play_button_should_be_displayed() and MainActivityTest#prev_and_next_buttons_should_be_disabled( ) get passed to Orchestrator and in 3. they are not found which causes the instrumentation to fail.

AndroidJUnitRunner code responsible for collecting tests:

I can see two ways to potentially fix it:

1. de.mannodermaus.junit5.AndroidJUnit5 could override getDescription and create something compliant with Test Orchestrator.
2. Create a custom AndroidJUnitRunner which would convert JUnit5 created class arguments to real tests (I’ve done something similar in https://github.com/stepstone-tech/AndroidTestXRunner/blob/master/xrunner-library/src/main/java/com/stepstone/xrunner/AndroidJUnitXRunner.kt). This has two downsides though… 1 - it’s super hacky and requires reflection, 2 - converting simple class names to full names is problematic and it means there should be no tests with the same name.

Please let me know what you think about it and if Android Test Orchestrator support is planned 😃

[mannodermaus]

I’m definitely going to add a section to the Wiki & the README file. I hope that we’ll find a solution that makes the Orchestrator work more transparently in the future, but for now, this is good enough. Thanks again for the research!

[zawadz88]

Yes, that’s exactly the issue. I’ll check with junit-platform-suite-api and post back here. Thanks for replying! P.S. if you’re interested in the Orchestrator you might check out this article: https://medium.com/stepstone-tech/android-test-orchestrator-unmasked-83b8879928fa it’s basically what I described here but in more detail