AGP 3.4.0 doesn't play with Android Test Orchestrator (not AndroidX version): ERROR: could not find class 'android.support.test.services.shellexecutor.ShellMain'

Description

I’m not sure if this is a Testing issue or an AGP issue.

Running instrumentation tests with AGP 3.4.0 and Test Orchestrator 1.0.2 causes process crash.

Steps to Reproduce

I have repro steps and a reproducible repo here https://github.com/tir38/android-test-orchestrator-bug

1. create brand new A/S project
2. see that AGP is set to ‘com.android.tools.build:gradle:3.4.0-beta05’
3. add Android Test Orchestrator to project (not the AndroidX version)
4. start up an emulator
5. run the instrumentation test that was included in brand new project:

$ ./gradlew clean app:connectedDebugAndroidTest

1. see the tests fail:

Task :app:connectedDebugAndroidTest FAILED

FAILURE: Build failed with an exception.

• What went wrong: Execution failed for task ‘:app:connectedDebugAndroidTest’. There were failing tests. See the report at: file:…/app/build/reports/androidTests/connected/index.html

1. open report and see:

No tests found. This usually means that your test classes are not in the form that your test runner expects (e.g. don’t inherit from TestCase or lack @Test annotations).

1. look at the logs on the emulator. see tombstone

JNI DETECTED ERROR IN APPLICATION: 
JNI NewGlobalRef called with pending exception java.lang.ClassNotFoundException: 
Didn't find class "android.support.test.services.shellexecutor.ShellMain" on path:...

Expected Results

Test should pass

Actual Results

Test process crashes

AndroidX Test and Android OS Versions

• Not on androidX yet
• Emulator running Android SDK 27

Link to a public git repo demonstrating the problem:

https://github.com/tir38/android-test-orchestrator-bug

[danielgomezrico]

@tir38 How did you declare the dependency?

In my case I had it:

androidTestImplementation ‘androidx.test:orchestrator:1.1.0’

but it should be:

androidTestUtil ‘androidx.test:orchestrator:1.1.0’

After I made this change it worked fine.

[nickpalmer]

Note that testing does work correctly with AndroidX Orchestrator. Changing the following two lines fixes things.

defaultConfig {
    testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
}
testOptions {
    execution 'ANDROIDX_TEST_ORCHESTRATOR'
}