TestNG swallowing an IllegalArgumentException when DataProivder mismatches Test

I have hit an issue where TestNG will hide a IllegalArgumentException when it throws one and the exception is marked as an expectedExceptions. This happens when TestNG is matching a dataProvider up to a test’s parameters, the data does not match up, and IllegalArgumentException is an expectedExceptions. An example of this is:

import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

public class swallowedException
{
    @DataProvider
    public Object[][] testInts()
    {
        return new Object[][] {
                new Object[] {new Integer(4) },
                new Object[] {new Integer(8) },
                new Object[] {new Integer(12) } };
    }

    @Test(dataProvider = "testInts", expectedExceptions = IllegalArgumentException.class)
    public void theTest(String aString)
    {
        System.out.println("An item is \"" + aString + "\"");
    }
}

In this case TestNG will throw an IllegalArgumentException when trying to match up the Integer from the dataProvider to the String parameter in the test (which is as expected.) However, because IllegalArgumentException is marked as an expected exception in the test, TestNG is marking this test as PASSED (all 3 times) when in fact it was not run. For this example, if you remove the expectedExceptions annotation, TestNG marks the test as FAILED with the exception thrown by TestNG. The actual code being tested was expected to throw an IllegalArgumentException, was not being tested, and was marked by TestNG as PASSED.

[dimo414]

@juherr I can’t promise anything, but I’ll certainly send a pull request if I have time.

[dimo414]

Looks like this is fixed, thanks! I think f3e44d60bf67 was the fix (since it added the “Data provider mismatch” message in DataProviderMethodMatcher). This should probably be called out in the release notes (if it’s not already) since it’ll cause some “passing” tests to start failing.