Test cases are skipped with TestCaseSource under Visual Studio 2019

Running latest versions of Visual Studio 2019 and NUnit (Visual studio 2019 16.3.5, NUnit 3.12.0, NUnit3TestAdapter 3.15.1)

Test with a test source:

[Test]
[TestCaseSource(nameof(TestCases))]
public async Task When_InvalidRequest_ReturnsValidationError(PlayerBalancesRequest request, string siteCode, string correlationToken)
{
   ...
}

public class PlayerBalancesRequest
{
    public long PlayerId { get; set; }
    public string Token { get; set; }
    public string CurrencyCode { get; set; }
}

private static object[] TestCases()
{
    return new object[]
    {
        new object[] { new PlayerBalancesRequest(), null, null },
        new object[] { new PlayerBalancesRequest { PlayerId = 0, Token = GameToken, CurrencyCode = CurrencyCode }, SiteCode, CorrelationToken },
        new object[] { new PlayerBalancesRequest { PlayerId = PlayerId, Token = null, CurrencyCode = CurrencyCode }, SiteCode, CorrelationToken },
        new object[] { new PlayerBalancesRequest { PlayerId = PlayerId, Token = GameToken, CurrencyCode = null}, SiteCode, CorrelationToken },
        ...
    };
}

I have 15 test cases in the test case source. Visual Studio shows all of them in the Test Explorer, but NUnit always runs just 3 of them. Always the same 3, and the rest are skipped. If I remove these 3, then it runs no tests. As a separate issue, if I leave just 1 test case in the test case source, it will think this object[3] that is returned are 3 separate test cases (PlayerBalancesRequest, string and string) and will run the test 3 times with just 1 parameter, failing because the test requires 3 parameters.

[OsirisTerje]

Yes, the use of `

TestContext.CurrentContext.Random.NextGuid()

also yields usable guids. I’ve pushed up code to the repo showing that too.

[CharliePoole]

You have only shown four of your test cases, but even with that small number, I can see that three of them will generate tests with the same name. That’s the source of your problem.

There’s a mismatch between who NUnit works and how TestExplorer wants to see tests. NUnit doesn’t care if multiple tests have the same name, because it doesn’t use the name to identify the tests. TestExplorer expects each test to have a unique name. The NUnit test adapter does nothing to ensure this. It merely passes on the name provided (usually implicitly) by the user.

In your case, the first argument PlayerBalanceRequest has a default string representation - IIRC <PlayerBalanceRequest>. The second and third args areinsufficient to provide uniqueness. For the four cases you give above, the two names generated will be something like the following, substituting the actual string values for SITECODE and TOKEN.

When_InvalidRequest_ReturnsValidationError(<PlayerBalanceRequest>, null, null)
When_InvalidRequest_ReturnsValidationError(<PlayerBalanceRequest>, "SITECODE", "TOKEN")

Generally, when tests have the same name, TestExplorer groups all the results under the first one found. Select one of the tests that ran and look at the result report, usually directly under the tree but sometimes next to it if your window size is small. I suspect you’ll find all fifteen.

So, what can you do about this? There are a two main alternatives…

1. Override PlayerBalanceRequest.ToString() so the representation includes the values of all the member properties. This will give you unique names for each case, so long as there are no duplicates - I assume you wouldn’t want duplicates anyway.

2. Replace the inner new object[]s with new TestCaseData() and use TestCaseData.SetName to assign a unique name to each case. If you do this, I advise replacing the outer new object[] with new TestCaseData[] because it’s much clearer. Personally, I believe that use of TestCaseData is a better practice in general, because it clearly says: “Here is one set of test case data!” and because you can easily add features like ignoring a case, changing the name, providing a description, etc.

Regarding the “separate issue” you mentioned, Did you eliminate the outer object[]? If you did that, it’s a user error. 😄 If you retained the nesting, then it might be a bug in the framework. In either case, the problem goes away if you use TestCaseData instead.