Extra test cases for DataRow methods

Description

After upgrading MSTest to MSTest v2, there is an extra test case for each DataRow method (like a summary to all data row tests of the method). And that causes the number of total tests, passed and failed are also not expected. It is fine with MSTest v1 and vstest.cosole.exe.

Is this a known issue or expected feature? Is there any parameters can be set to change the behavior?

Steps to reproduce

    [DataTestMethod]
    [DataRow("1")]
    [DataRow("2")]
    public void TestMethodA(string a)
    {
        Assert.IsTrue(true);
    }

    [DataTestMethod]
    [DataRow("1")]
    [DataRow("2")]
    public void TestMethodB(string a)
    {
        Assert.IsTrue(a.Equals("1"));
    }

Expected behavior

Passed   TestMethodA (Data Row 0)
Passed   TestMethodA (Data Row 1)
Passed   TestMethodB (Data Row 0)
Failed   TestMethodB (Data Row 1)
Total tests: 4. Passed: 3. Failed: 1. Skipped: 0.

Actual behavior

Passed   TestMethodA
Passed   TestMethodA (Data Row 0)
Passed   TestMethodA (Data Row 1)
Failed   TestMethodB
Passed   TestMethodB (Data Row 0)
Failed   TestMethodB (Data Row 1)
Total tests: 6. Passed: 4. Failed: 2. Skipped: 0.

Environment

Run tests through vstest.console.exe on Azure DevOps MSTest 1.3.2

[enisak]

2.2.8, works totally fine for me

[Hylke-Atmos]

I’d like to argue that this is a bug and undesired behavior!

It seems that not only the results seem off. It also seems like it is incorrectly dealing with the DynamicData method used for retrieving the data. The first entry is retrieved twice (first for the initial call and then second time for actual execution of the data).

Due to this behavior we have a test that is simply failing because its data (a FileStream) is retrieved/opened twice which is (by default) not allowed by the file system.

It is not the most pretty test, but it is a simple and effective test for checking the function that is verifying the version we embed in a file.

public static IEnumerable<object[]> GetFileStreamForGetVersionFromFileStream()
{
    yield return new object[] { new FileStream(@"Assets\temp1.x", FileMode.Open), 0x01 };
    yield return new object[] { new FileStream(@"Assets\temp2.x", FileMode.Open), 0x02 };
    yield return new object[] { new FileStream(@"Assets\temp3.x", FileMode.Open), 0x03 };
}

[DataTestMethod]
[DynamicData(nameof(GetFileStreamForGetVersionFromFileStream), DynamicDataSourceType.Method)]
public void GetVersionFromFileStreamReturnsCorrectValue(FileStream fs, int expected)
{
   MethodInfo method =
       typeof(FileHandler).GetMethod("GetVersionFromFileStream", BindingFlags.NonPublic | BindingFlags.Static);
   var actual = uint.MaxValue;
   if (method != null)
   {
       actual = (uint) method.Invoke(null, new object[] {fs});
   }
   Assert.AreEqual((uint)expected, actual);
   //Cleanup
   fs.Close();
}

This test fails because temp1.x is opened twice and causing the following exception:

The process cannot access the file '...\Assets\temp1.x' because it is being used by another process.

Although it is possible to overcome this particular problem by using the FileShare.Read flag, it is undesired (and unexpected) behavior that MSTest is retrieving and locking the resources twice!