Analyzing unreliable CI

I tried to get any clue on whats going on.

At this moment and over the last days the executors cgeo-1 and cgeo-5 have been online and serving the PR and commit jobs.

Looking at cgeo-5:

It seems to fail most (but not all) builds with problems to install the APK on the emulator:

00:56:08  Installing APK 'cgeo-debug.apk' on 'android-30(AVD) - 11' for main:basic-debugUnable to install /home/jenkins/slave/workspace/cgeo CI - PR build/main/build/outputs/apk/basic/debug/cgeo-debug.apk
00:56:08  
00:56:08  com.android.ddmlib.InstallException: Unknown failure: Exception occurred while executing 'install':
00:56:08  java.lang.IllegalArgumentException: Error: Failed to parse APK file: /data/local/tmp/cgeo-debug.apk: Failed to parse /data/local/tmp/cgeo-debug.apk
00:56:08  at com.android.server.pm.PackageManagerShellCommand.setParamsSize(PackageManagerShellCommand.java:528)
00:56:08  at com.android.server.pm.PackageManagerShellCommand.doRunInstall(PackageManagerShellCommand.java:1282)
00:56:08  at com.android.server.pm.PackageManagerShellCommand.runInstall(PackageManagerShellCommand.java:1248)
00:56:08  at com.android.server.pm.PackageManagerShellCommand.onCommand(PackageManagerShellCommand.java:184)
00:56:08  at android.os.BasicShellCommandHandler.exec(BasicShellCommandHandler.java:98)
00:56:08  at android.os.ShellCommand.exec(ShellCommand.java:44)
00:56:08  at com.android.server.pm.PackageManagerService.onShellCommand(PackageManagerService.java:21646)
00:56:08  at android.os.Binder.shellCommand(Binder.java:929)
00:56:08  at android.os.Binder.onTransact(Binder.java:813)
00:56:08  at android.content.pm.IPackageManager$Stub.onTransact(IPackageManager.java:4603)
00:56:08  at com.android.server.pm.PackageManagerService.onTransact(PackageManagerService.java:4305)
00:56:08  at android.os.Binder.execTransactInternal(Binder.java:1159)
00:56:08  at android.os.Binder.execTransact(Binder.java:1123)
00:56:08  Caused by: java.io.FileNotFoundException: AndroidManifest.xml
00:56:08  at android.content.res.ApkAssets.nativeOpenXml(Native Method)
00:56:08  at android.content.res.ApkAssets.openXml(ApkAssets.java:367)
00:56:08  at android.content.pm.parsing.ApkLiteParseUtils.parseApkLiteInner(ApkLiteParseUtils.java:245)
00:56:08  at android.content.pm.parsing.ApkLiteParseUtils.parseApkLite(ApkLiteParseUtils.java:226)
00:56:08  at com.android.server.pm.PackageManagerShellCommand.setParamsSize(PackageManagerShellCommand.java:522)
00:56:08  ... 12 more

I have no idea why this is happening? Looks like either the APK is not present at the source or does not contain what should be in there (See: java.io.FileNotFoundException: AndroidManifest.xml)

FYA: I excluded cgeo-5 from job execution by removing its labels (has-emulator has-emulator2) in its preferences.

Looking at cgeo-1:

This executors does at least seem not to show the installation problems of cgeo-5. Looking at its build history it seems instead having sporadic problems, typically this failing initial test:

11:15:07  cgeo.geocaching.CgeoApplicationTest > testPreconditions[android-30(AVD) - 11] FAILED 
11:15:07  	org.junit.ComparisonFailure: [User and password must be provided] expected:<[NO_ERROR]> but was:<[CONNECTION_FAILED]>
11:15:07  	at java.lang.reflect.Constructor.newInstance0(Native Method)
11:25:52  Build timed out (after 14 minutes). Marking the build as aborted.
11:25:52  FATAL: Caught exception while loading script 'cgeo build and test for new commits and pull requests v2'

Not sure, but looks like a problem to reach the server or problem with internet connection in general? There are other sporadic failures on cgeo-1 as well, which I cannot judge as it also might be correct fails.

cgeo-5 is still excluded from building right now (see above). Lets continue to use cgeo-1 to get a clear picture and try to fix that one first.

@mucek4 Can you summarize here:

• Are cgeo-1 and cgeo-5 running on the same hardware?
• Could it be a internet connection or general network problem on cgeo-1?

[kumy]

🍾

Success Build #305 (Nov 28, 2021, 9:22:16 AM)
PR #12141: WIP: Testing GKv2 website o...
Took 9 min 21 sec on kumy-1

[kumy]

@Lineflyer

After hours with manual builds git bisecting, scratching and banging my head on the table, I’m glad to announce I’ve finally get something quite stable, at least for the last 8 builds (starting from #3​28)…

Don’t ask me why, but after having decoupled the old build line to small steps, it seems to work more reliably:

# From
./gradlew --no-daemon --scan clean testDebug createBasicDebugCoverageReport checkstyle lintBasicDebug -Pandroid.testInstrumentationRunnerArguments.notAnnotation=cgeo.geocaching.test.NotForIntegrationTests

---

# To
./gradlew --no-daemon --scan  clean
rm -fr build build-cache main/build || true

./gradlew --no-daemon --scan  testDebug -Pandroid.testInstrumentationRunnerArguments.notAnnotation=cgeo.geocaching.test.NotForIntegrationTests
./gradlew --no-daemon --scan  checkstyle
./gradlew --no-daemon --scan  lintBasicDebug

#Note, removed `createBasicDebugCoverageReport` as this target, doesn't exists anymore?

From what I saw, the main problem was the apk generation. The resulting file is normally ~17M, but seen it at 2.2M, 10M when it fails. The AndroidManifest.xml, and all res/ folder is missing.

Maybe some graddle guru can figure out why this little guy has problem 😃

“Required” changes are applied to script https://ci.cgeo.org/configfiles/editConfig?id=0591d16e-bf19-43f3-9421-af26d4d45276, Also added some Failure causes in Failure Cause Management

Executors cgeo-1 cgeo-2 are back online (tag re-added) + I let kumy-1 up some days.

I really hope it will stay stable 🤞 🤞 🤞 🤞 🤞

Edit: Also updated the cgeo-executor a bit on branch v5