Azure DevOps test result publishing failures

We’re seeing failures, sometimes with errors as below, sometimes not, where the test runs created don’t seem to exist. We have tweaked some settings with respect to the system access token recently; this may be related, but I’ll likely open an IcM to track this on the AzDO side shortly if this can’t be attributed to that.

Representative instance of the problem: https://github.com/dotnet/roslyn/pull/61351 From a representative log:

2022-05-17T00:35:20.336Z	ERROR  	azure_devops_result_publisher(134)	log_error	got error: Traceback (most recent call last):
  File "C:\h\scripts\helix-scripts\helix\test_reporting\azure_devops_result_publisher.py", line 139, in upload
    self._process(batch)
  File "C:\h\scripts\helix-scripts\helix\test_reporting\azure_devops_result_publisher.py", line 151, in _process
    hot_path_tests += self._publish_results(converted)
  File "C:\h\scripts\helix-scripts\helix\test_reporting\azure_devops_result_publisher.py", line 287, in _publish_results
    published_results = self._send(lambda: test_client.add_test_results_to_test_run(
  File "C:\h\scripts\helix-scripts\helix\test_reporting\azure_devops_result_publisher.py", line 265, in _send
    raise ex
  File "C:\h\scripts\helix-scripts\helix\test_reporting\azure_devops_result_publisher.py", line 241, in _send
    return execute()
  File "C:\h\scripts\helix-scripts\helix\test_reporting\azure_devops_result_publisher.py", line 287, in <lambda>
    published_results = self._send(lambda: test_client.add_test_results_to_test_run(
  File "C:\h\scripts\helix-scripts\azure\devops\v5_1\test\test_client.py", line 675, in add_test_results_to_test_run
    response = self._send(http_method='POST',
  File "C:\h\scripts\helix-scripts\azure\devops\client.py", line 104, in _send
    response = self._send_request(request=request, headers=headers, content=content, media_type=media_type)
  File "C:\h\scripts\helix-scripts\azure\devops\client.py", line 68, in _send_request
    self._handle_error(request, response)
  File "C:\h\scripts\helix-scripts\azure\devops\client.py", line 247, in _handle_error
    raise AzureDevOpsServiceError(wrapped_exception)
azure.devops.exceptions.AzureDevOpsServiceError: Test result 0 of test run 47634538 cannot be found.
Traceback (most recent call last):
  File "C:\h\scripts\helix-scripts\helix\test_reporting\azure_devops_result_publisher.py", line 139, in upload
    self._process(batch)
  File "C:\h\scripts\helix-scripts\helix\test_reporting\azure_devops_result_publisher.py", line 151, in _process
    hot_path_tests += self._publish_results(converted)
  File "C:\h\scripts\helix-scripts\helix\test_reporting\azure_devops_result_publisher.py", line 287, in _publish_results
    published_results = self._send(lambda: test_client.add_test_results_to_test_run(
  File "C:\h\scripts\helix-scripts\helix\test_reporting\azure_devops_result_publisher.py", line 265, in _send
    raise ex
  File "C:\h\scripts\helix-scripts\helix\test_reporting\azure_devops_result_publisher.py", line 241, in _send
    return execute()
  File "C:\h\scripts\helix-scripts\helix\test_reporting\azure_devops_result_publisher.py", line 287, in <lambda>
    published_results = self._send(lambda: test_client.add_test_results_to_test_run(
  File "C:\h\scripts\helix-scripts\azure\devops\v5_1\test\test_client.py", line 675, in add_test_results_to_test_run
    response = self._send(http_method='POST',
  File "C:\h\scripts\helix-scripts\azure\devops\client.py", line 104, in _send
    response = self._send_request(request=request, headers=headers, content=content, media_type=media_type)
  File "C:\h\scripts\helix-scripts\azure\devops\client.py", line 68, in _send_request
    self._handle_error(request, response)
  File "C:\h\scripts\helix-scripts\azure\devops\client.py", line 247, in _handle_error
    raise AzureDevOpsServiceError(wrapped_exception)
azure.devops.exceptions.AzureDevOpsServiceError: Test result 0 of test run 47634538 cannot be found.

{
   "ErrorMessage" : "has one or more failing tests based on run statistics, but I couldn't find the failures."
}

Report

Build	Definition	Step Name	Console log
1777488	dotnet/runtime	Send to Helix	https://dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_apis/build/builds/1777488/logs/1232
1777179	dotnet/runtime	Send to Helix	https://dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_apis/build/builds/1777179/logs/1261

Summary

Day Hit Count	Week Hit Count	Month Hit Count
0	2	2

[MattGal]

The tests pane on almost every runtime run going quite far back is missing many of the (passing) run results. AzDO has acknowledged the issue and is working on a fix. It’s great you’ve figured this out, but I’m still going to keep this issue around until that’s resolved.

[Neme12]

@MattGal I think it has to do with the full name of the test being too long - the name property in the results file, which includes the namespace, class name, method name and all parameters:

<test name="Microsoft.CodeAnalysis.UnitTests.WorkspaceServiceTests.DirectMemoryAccessStreamReaderTests.ReadToArray(bufferLength: 10, index: 3, count: 7, expected: &quot;\0\0\0bcdefgh&quot;, expectedResult: 7, expectedPeek: -1)" type="Microsoft.CodeAnalysis.UnitTests.WorkspaceServiceTests.DirectMemoryAccessStreamReaderTests" method="ReadToArray" time="0.0010000" result="Pass">
  <traits>
    <trait name="Feature" value="Workspace" />
  </traits>
</test>

I tried doing various thing in a new PR (https://github.com/dotnet/roslyn/pull/61363) to get the problematic tests to succeed - if I removed all the tests I added, it succeeded, I added more and more tests and it still succeeded, until I added the one that has a lot of parameters - then it failed. After that, I tried shortening the namespace and class name and now it succeeded. Then I restored the namespace to the longer version and it failed. This points to the fully qualified test name being too long.