TestId no longer constant across test runs

Problem:

Trx specification seems to have changed significantly from 2.2.3 to 2.2.4+. As a result it makes it difficult (impossible?) to track historic results, map child tests to parents, and muddies the water for all UnitTestResults in the Trx file when using DataRow and DynamicData attributes.

In 2.2.3 all DataRow and DynamicData tests would show up as an InnerResult for a parent. The result would contain a testId that mapped to the parent’s testId that it is nested under. This made sense because you could see the overall test result by viewing the parent and then look at each individual test result when looking at the child tests. After upgrading to 2.2.4 or later it doesn’t seem like any of the consistency that Trx files had are valid anymore.

This has negative side effects depending on the CI/CD platform you use. In our example we use Bamboo which has always displayed parent results. Now that we have upgraded it shows each individual DataRow and DynamicData test result now too. While that might be okay for teams using default tooling that doesn’t require any custom plugins or features to maintain a code base these new changes have made it difficult for those of us who need extra feature sets to keep our teams sailing smoothly.

Our team has a monolithic repository. Unit, integration, and functional tests can all be found in our C# test projects that run as part of our continuous integration process. As a result we have had to build tooling to manage and maintain tests. To do this we process Trx files, store them in our own data store, and run analysis on them. Some reasons why are listed below.

• Does the test have the traits of a flaky test? If so, quarantine it in Bamboo so developers are not blocked from merging.
• Is this test too slow consistently? If so, notify users that there is a reproducible performance issue.
• Is this test randomly slow? If so, notify users that there is a hard to find performance issue.
• Does this test fail the first time but pass on the second attempt? If so, notify users that their test is unreliable.
• Does the test only fail between certain hours of the day? If so, notify the user.

You can see where that is going.

To create such a system we have had to rely on constants between test runs that lets us map results historically. For the last 8 years the test id has always been that constant. You couldn’t even open a Trx file in Visual Studio if there were duplicate test ids being used. I haven’t tested that in a while though.

If we want to quarantine a test we could quarantine the parent and that would impact the children. Under this Trx structure in 2.2.4+ that is no longer the case. While it would be fine to quarantine based on the ClassName+TestName we are noticing that the ClassName+TestName is not unique across all test results either. Example of that below:

<?xml version="1.0" encoding="utf-8"?>
<TestRun id="8ace4773-a85f-4063-b352-b47056691b99" name="" runUser="" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Times creation="2022-07-13T20:02:39.7924341-04:00" queuing="2022-07-13T20:02:39.7924341-04:00" start="2022-07-13T20:02:36.5608957-04:00" finish="2022-07-13T20:02:39.8334322-04:00" />
  <TestSettings name="default" id="97324b85-62c7-41b6-9b86-22bf059b57dd">
    <Deployment runDeploymentRoot="" />
  </TestSettings>
  <Results>
    <UnitTestResult executionId="45342629-aefa-427e-9ca0-c94b9cf906fe" testId="28a880b7-582f-47a1-a6dd-b2b105ee4be9" testName="DyanmicDataExample_SameSignaturePerTestRun (TrxTesting.EnergyProgram,TrxTesting.Address,True)" computerName="WCJEWETHQ01W05" duration="00:00:00.0000013" startTime="2022-07-13T20:02:39.3124340-04:00" endTime="2022-07-13T20:02:39.6204751-04:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="45342629-aefa-427e-9ca0-c94b9cf906fe" />
    <UnitTestResult executionId="ef2df86d-58f8-4bc9-940c-fae0153f0128" testId="1c3e44a0-7941-4aef-a4c6-ea3d00b7e7be" testName="DyanmicDataExample_SameSignaturePerTestRun (TrxTesting.EnergyProgram,TrxTesting.Address,False)" computerName="WCJEWETHQ01W05" duration="00:00:00.0000014" startTime="2022-07-13T20:02:39.3124340-04:00" endTime="2022-07-13T20:02:39.6204751-04:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="ef2df86d-58f8-4bc9-940c-fae0153f0128" />
    <UnitTestResult executionId="7a3aa101-0b49-4b39-8f19-36f8e5b43d04" testId="a8fb6105-cb12-451c-a219-e085bf9a66dd" testName="DyanmicDataExample_SameSignaturePerTestRun (TrxTesting.EnergyProgram,TrxTesting.Address,True)" computerName="WCJEWETHQ01W05" duration="00:00:00.0000125" startTime="2022-07-13T20:02:39.3124340-04:00" endTime="2022-07-13T20:02:39.6204751-04:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="7a3aa101-0b49-4b39-8f19-36f8e5b43d04" />
    <UnitTestResult executionId="3fb73645-1469-48a0-b24e-488e17b9aed4" testId="1275e361-7c97-4d0d-86a9-504c6d127622" testName="DyanmicDataExample_SameSignaturePerTestRun (TrxTesting.EnergyProgram,TrxTesting.Address,False)" computerName="WCJEWETHQ01W05" duration="00:00:00.0000013" startTime="2022-07-13T20:02:39.3124340-04:00" endTime="2022-07-13T20:02:39.6204751-04:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="3fb73645-1469-48a0-b24e-488e17b9aed4" />
    <UnitTestResult executionId="b15f150b-7490-41da-9723-fff035ea8788" testId="ab955369-be12-465c-a81e-b0817cc2cdbb" testName="DyanmicDataExample_SameSignaturePerTestRun (,TrxTesting.Address,False)" computerName="WCJEWETHQ01W05" duration="00:00:00.0083030" startTime="2022-07-13T20:02:39.3124340-04:00" endTime="2022-07-13T20:02:39.6204751-04:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="b15f150b-7490-41da-9723-fff035ea8788" />
    <UnitTestResult executionId="0e470c2e-cffd-4eab-8348-6aa0e87a8317" testId="301a572a-19ae-40a2-abd8-f855835e8d7c" testName="DyanmicDataExample_SameSignaturePerTestRun (TrxTesting.EnergyProgram,TrxTesting.Address,True)" computerName="WCJEWETHQ01W05" duration="00:00:00.0000020" startTime="2022-07-13T20:02:39.3124340-04:00" endTime="2022-07-13T20:02:39.6204751-04:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="0e470c2e-cffd-4eab-8348-6aa0e87a8317" />
  </Results>
  <TestDefinitions>
    <UnitTest name="DyanmicDataExample_SameSignaturePerTestRun (TrxTesting.EnergyProgram,TrxTesting.Address,False)" storage="c:\users\cjewett\source\trxtesting\bin\debug\net48\trxtesting.dll" id="1275e361-7c97-4d0d-86a9-504c6d127622">
      <Execution id="3fb73645-1469-48a0-b24e-488e17b9aed4" />
      <TestMethod codeBase="C:\Users\cjewett\source\TrxTesting\bin\Debug\net48\TrxTesting.dll" adapterTypeName="executor://mstestadapter/v2" className="TrxTesting.UnitTest1" name="DyanmicDataExample_SameSignaturePerTestRun" />
    </UnitTest>
    <UnitTest name="DyanmicDataExample_SameSignaturePerTestRun (TrxTesting.EnergyProgram,TrxTesting.Address,True)" storage="c:\users\cjewett\source\trxtesting\bin\debug\net48\trxtesting.dll" id="a8fb6105-cb12-451c-a219-e085bf9a66dd">
      <Execution id="7a3aa101-0b49-4b39-8f19-36f8e5b43d04" />
      <TestMethod codeBase="C:\Users\cjewett\source\TrxTesting\bin\Debug\net48\TrxTesting.dll" adapterTypeName="executor://mstestadapter/v2" className="TrxTesting.UnitTest1" name="DyanmicDataExample_SameSignaturePerTestRun" />
    </UnitTest>
    <UnitTest name="DyanmicDataExample_SameSignaturePerTestRun (TrxTesting.EnergyProgram,TrxTesting.Address,True)" storage="c:\users\cjewett\source\trxtesting\bin\debug\net48\trxtesting.dll" id="301a572a-19ae-40a2-abd8-f855835e8d7c">
      <Execution id="0e470c2e-cffd-4eab-8348-6aa0e87a8317" />
      <TestMethod codeBase="C:\Users\cjewett\source\TrxTesting\bin\Debug\net48\TrxTesting.dll" adapterTypeName="executor://mstestadapter/v2" className="TrxTesting.UnitTest1" name="DyanmicDataExample_SameSignaturePerTestRun" />
    </UnitTest>
    <UnitTest name="DyanmicDataExample_SameSignaturePerTestRun (TrxTesting.EnergyProgram,TrxTesting.Address,False)" storage="c:\users\cjewett\source\trxtesting\bin\debug\net48\trxtesting.dll" id="1c3e44a0-7941-4aef-a4c6-ea3d00b7e7be">
      <Execution id="ef2df86d-58f8-4bc9-940c-fae0153f0128" />
      <TestMethod codeBase="C:\Users\cjewett\source\TrxTesting\bin\Debug\net48\TrxTesting.dll" adapterTypeName="executor://mstestadapter/v2" className="TrxTesting.UnitTest1" name="DyanmicDataExample_SameSignaturePerTestRun" />
    </UnitTest>
    <UnitTest name="DyanmicDataExample_SameSignaturePerTestRun (TrxTesting.EnergyProgram,TrxTesting.Address,True)" storage="c:\users\cjewett\source\trxtesting\bin\debug\net48\trxtesting.dll" id="28a880b7-582f-47a1-a6dd-b2b105ee4be9">
      <Execution id="45342629-aefa-427e-9ca0-c94b9cf906fe" />
      <TestMethod codeBase="C:\Users\cjewett\source\TrxTesting\bin\Debug\net48\TrxTesting.dll" adapterTypeName="executor://mstestadapter/v2" className="TrxTesting.UnitTest1" name="DyanmicDataExample_SameSignaturePerTestRun" />
    </UnitTest>
    <UnitTest name="DyanmicDataExample_SameSignaturePerTestRun (,TrxTesting.Address,False)" storage="c:\users\cjewett\source\trxtesting\bin\debug\net48\trxtesting.dll" id="ab955369-be12-465c-a81e-b0817cc2cdbb">
      <Execution id="b15f150b-7490-41da-9723-fff035ea8788" />
      <TestMethod codeBase="C:\Users\cjewett\source\TrxTesting\bin\Debug\net48\TrxTesting.dll" adapterTypeName="executor://mstestadapter/v2" className="TrxTesting.UnitTest1" name="DyanmicDataExample_SameSignaturePerTestRun" />
    </UnitTest>
  </TestDefinitions>
  <TestEntries>
    <TestEntry testId="28a880b7-582f-47a1-a6dd-b2b105ee4be9" executionId="45342629-aefa-427e-9ca0-c94b9cf906fe" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
    <TestEntry testId="1c3e44a0-7941-4aef-a4c6-ea3d00b7e7be" executionId="ef2df86d-58f8-4bc9-940c-fae0153f0128" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
    <TestEntry testId="a8fb6105-cb12-451c-a219-e085bf9a66dd" executionId="7a3aa101-0b49-4b39-8f19-36f8e5b43d04" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
    <TestEntry testId="1275e361-7c97-4d0d-86a9-504c6d127622" executionId="3fb73645-1469-48a0-b24e-488e17b9aed4" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
    <TestEntry testId="ab955369-be12-465c-a81e-b0817cc2cdbb" executionId="b15f150b-7490-41da-9723-fff035ea8788" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
    <TestEntry testId="301a572a-19ae-40a2-abd8-f855835e8d7c" executionId="0e470c2e-cffd-4eab-8348-6aa0e87a8317" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
  </TestEntries>
  <TestLists>
    <TestList name="Results Not in a List" id="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
    <TestList name="All Loaded Results" id="19431567-8539-422a-85d7-44ee4e166bda" />
  </TestLists>
  <ResultSummary outcome="Completed">
    <Counters total="6" executed="6" passed="6" failed="0" error="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />
  </ResultSummary>
</TestRun>

Under our current system we store these results as results under the parent and perform our analysis that way. If we detect an issue we tell them the parent is problematic and they should take a look. Upgrading to 2.2.4+ forces us to have to process these child test results as their own parent because that is how the Trx is now structured. The problem is there is no consistent unique id to map results to now.

The TestId changes every test run.

The ClassName+TestName can have multiple tests that are not the exact same test but have the same signature. Therefore you store different test results under a single ClassName+TestName signature which would be incorrect.

The ClassName+TestName is also random at times depending on the type of data being passed in. For example, if you pass in DateTime.UtcNow then the ClassName+TestName (and TestId) signature should change every test run. The saving grace in 2.2.3 or below is that all of these results were stored under a parent, and the parent had a constant TestId.

We can’t ignore processing of these DynamicData test results either because the parent test result no longer exists in the Trx file. In addition to that it brings on more trouble of trying to figure out what kind of ClassName+TestName pattern you ignore. Also, if we don’t provide support for processing these results then we cannot quarantine them as part of our continuous integration process and that hurts our development process.

Here is the code used to replicate these problems: https://github.com/Cjewett/TrxTestIdIssue/tree/main

All that said it is possible we have missed something that still allows us to track results historically across different test runs, so we had some questions and were hoping for guidance.

• Is the TestId really supposed to be changing every test run now?
• Is it expected that multiple test results can have the same ClassName+TestName signature?
• Is there a way to revert back to the old Trx format using the new library versions?
• Does Microsoft consider any of this a bug? In other threads it seems like others have run into issues and it’s odd that the Trx data changed in this manner under a minor version bump.

Any guidance and information so we can make a decision on how to move forward is much appreciated. Thanks!

AB#1579034

[Evangelink]

@Cjewett Sorry it’s taking us so long in handling your request but I am actively working on it now. In the PR linked, we are introducing some enum that allows to select the strategy used to generate the test ID.

A couple of side notes:

• There were some issues on our side that lead to new features and breaking changes to be added in patch versions (2.2.4+).
• The test ID change introduced in 2.2.4 relies on Display Name and doesn’t take into account data so we have a couple of open issues

Is the TestId really supposed to be changing every test run now?

In all strategies, I don’t expect the test ID to change between run as long as the file path remains the same. From my investigations, it seems that the file path is used as part of the ID generation to act as some kind of discriminator in metadata changes between 2 dlls (e.g. Debug and Release build would produce 2 dlls in different locations so we need to consider the tests as different).

Is it expected that multiple test results can have the same ClassName+TestName signature?

I would tend to say yes but it seems that xUnit and NUnit have different strategies here (xUnit seems to be removing duplicates while NUnit keeps them). I am still trying to be sure of what was the intent of MSTest on this one.

Is there a way to revert back to the old Trx format using the new library versions?

Changing the test ID generation strategy would help there BUT you will possibly revert back to broken “state” for some of the parameterized tests.

Does Microsoft consider any of this a bug? In other threads it seems like others have run into issues and it’s odd that the Trx data changed in this manner under a minor version bump.

We do consider this as a bug and we are trying to re-engage with community on this repo.

[Haplois]

TestId should stay constant, let me investigate this.