Consider adding check for completed state/postProcessSTate in CheckAzurePipelinesTestResults

Filed from Teams channel report by @steveisok

This failed test should have caused the build to fail due to this check in CheckAzurePipelinesTestResults

Checking out the test result API, https://dev.azure.com/dnceng-public/public/_apis/test/runs/2327350?api-version=6.0 has expected times for completed date, and (currently) contains the failed value that should have caused the build task to fail.

"startedDate": "2022-12-12T08:41:33.28Z",
"completedDate": "2022-12-12T09:11:19.437Z",

Reading the binlog, the CheckAzurePipeLinesTestResults task starts less than one second later, at 2022-12-12 09:11:20.0276219 indicating that the API call completing the run succeeded.

From the test reporting logs the reporting happens at 2022-12-12T09:02:02.551Z, a full 9 minutes before any checking for the run being in a failed state.

This is definitely going to be very difficult to show AzDO a live repro of, but perhaps we should simply check for “state” == “Completed” and “postProcessState” == “Complete”, and either poll and wait, or throw an exception if this is not the case inside CheckAzurePipelinesTestResults

[MattGal]

@steveisok After further investigation, the three Helix runs in the job you found (“8258dc95-45e7-4229-96b9-9346cb417ef2”,“91738bad-54c3-477a-9087-fc8e60cbc834”,“afc22ef0-1dc2-486a-9f7b-916c7241d2a6”) are the only in the last 30 days that have hit this symptom.

As such, adding detection and retry for incomplete runs (linked pr) and occasionally checking in on the below query should be all we need; this is a very unique occurrence and represents a failure in the Azure Devops API. It doesn’t seem like it could happen often enough to cause any lasting lack of test coverage.

For posterity, the query used to determine this was (in 5-day intervals since it hits memory limits):

let testFailureJobs = 
TestResults
| where ingestion_time() > ago(30d)
| where ingestion_time()  < ago(1d)
| where Result == "Fail"
| summarize by JobId;
let AzDoBuildIds =
Jobs | where ingestion_time() > ago(30d) 
| where ingestion_time()  < ago(1d)
| where JobId  in (testFailureJobs)
| extend props = parse_json(Properties)
| where props["Project"] == "public"
| extend identifier = strcat("Build.", props["System.PhaseName"], ".", props["System.JobName"])
| summarize by BuildId=toint(props["BuildId"]), identifier, JobId, Attempt=toint(props["System.PhaseAttempt"]);
cluster('1es.kusto.windows.net').database('AzureDevOps').BuildTimelineRecord
| where FinishTime > ago(30d) and FinishTime < ago(25d)
| where OrganizationName == "dnceng-public"
| where ProjectId == "cbb18261-c48f-4abb-8651-8cdcb5474649" // Public
| where Result == "succeeded"
| extend jData = parse_json(Data)
| extend identifier = tostring(jData["identifier"])
| join kind = inner (AzDoBuildIds) on BuildId, identifier, Attempt

[MattGal]

arcade-validation is clean post-merge, closing issue.