Timed out datarow affects subsequent datarows

Description

When executing a test with multiple datarows, and one datarow times out, all subsequent datarows are aborted and marked failed.

Steps to reproduce

Runt the following test:

    [TestClass]
    public class UnitTest1
    {
        [DataTestMethod]
        [DataRow(1)]
        [DataRow(2)]
        [DataRow(3)]
        [DataRow(2000)]
        [DataRow(4)]
        [DataRow(5)]
        [Timeout(500)]
        public async Task TestDataRowTimeout(int delay)
        {
            await Task.Delay(delay);
        }
    }

Expected behavior

Only datarow with value 2000 times out. However row 4 and 5 are aborted with message Test 'TestDataRowTimeout' execution has been aborted. The timeout doesn’t apply to the combined time for all rows - that can be confirmed by setting the first few rows to be very close to 500, and added up exeeding the timeout value. If the combined time spent mattered, it would time out before making it to the slow row.

Actual behavior

all rows after the timed out row are aborted.

Environment

VS2019.7. .NET Core 3.1. using latest packages.

.csproj used:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>

    <IsPackable>false</IsPackable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.7.0" />
    <PackageReference Include="MSTest.TestAdapter" Version="2.1.2" />
    <PackageReference Include="MSTest.TestFramework" Version="2.1.2" />
  </ItemGroup>

</Project>

[Haplois]

@dotMorten, this should be fixed with out latest release. Please use 16.10.0 version of Microsoft.NET.Test.Sdk and the latest preview of MSTest; and let us know if you still have this problem.

[nohwnd]

@dotMorten

I’m confused about the above response. It seems to deal with how to deal with tearing down a test after a timeout, and not with the fact that each datarow isn’t run as a new test but the outcome of the previous test can affect the outcome of the following tests.

There are multiple things mixed together.

1. MSTest (until the update you are testing), reported each testcase in the same test. The timeout applies to the test, that is to all testcases combined. If all testcases together take more time than the timeout, the timeout will be triggered.

2. With the update each test case is reported as a single test, and will get its own timeout.

3. BUT still, if any test times out, the timing out should trigger killing the process, which used to happen, but it seems to not happen anymore. Instead we complete a task that reports error, but the task that is running the test code still continues. It behaves the same way that NUnit is behaving. But now after the update the next test gets the chance to run, even though the timed out test is still running in the background.

using Microsoft.VisualStudio.TestTools.UnitTesting;
using System;
using System.Threading.Tasks;

namespace TestProject12
{
    [TestClass]
    public class UnitTest1
    {
        [DataRow("tc1", 500)]
        [DataRow("tc2", 2000)]
        [DataRow("tc3", 3000)]
        [Timeout(1000)]
        [TestMethod]
        public async Task Test(string name, int milliseconds)
        {
            System.IO.File.AppendAllLines(@"C:\t\mstest.txt", new[] { $"{name} start" });
            await Task.Delay(milliseconds);
            System.IO.File.AppendAllLines(@"C:\t\mstest.txt", new[] { $"{name} end" });
        }
    }

    [TestClass]
    public class UnitTest2
    {
        [TestMethod]
        public async Task Test2()
        {
            await Task.Delay(3000);
        }
    }
}

Output:

tc1 start
tc1 end
tc2 start
tc3 start
tc3 end
tc2 end
tc1 start
tc1 end
tc2 start
tc3 start
tc3 end
tc2 end

Pre update:

After update: