`ue4 test` doesn't exit with non-zero with failed tests

We recently noticed that we had passing builds with failing tests.

It turns out we were relying on ue4 test as the last command in our pipeline, and it fails to detect the test failures.

We do see this:

[2020.03.10-03.56.20:514][204]LogAutomationCommandLine: Display: Setting GIsCriticalError due to test failures (will cause non-zero exit code).

so UE4 is exiting with error code 1.

None of the strings checked in automationTests appear in our logs though.

The most-likely candidate for a string-based check looks like:

[2020.03.10-03.56.00:819][ 32]LogAutomationController: Error: Test Completed. Result={Failed} Name={MatchmakingTest} Path={Project.Functional Tests./PluginSystem/Tests/TestLevel.OneTest}

This is with the ue4-full Docker image for 4.24.2, so it’s possible the output format has changed in recent UE4 versions. I don’t know off-hand if this was also failing in 4.22.2, which we moved from a couple of weeks ago.

It would probably be nicer if this (and in fact all command wrappers) propagated a failure error code back to the caller, and then we could just use that.

[adamrehn]

Earlier versions of the Unreal Engine would yield an exit code of zero in the event of an automation test failure, which is why the string checking is performed to detect failures. I’ll need to run some tests with various versions of UE4 to determine the minimum version that provides accurate exit codes for propagation.

[TBBle]

I didn’t try a known-failing test, but the Linux auto-test run without failures now passes CI with 0.0.44, thank you.