Execution of test methods in different classes is interleaved when dependsOnMethod is used

I am seeing an odd order of test executions in TestNG.

I created a simple project, with this testng.xml:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="Simple suite">
    <test name="Simple tests">
        <packages>
            <package name="testngtests"></package>
        </packages>
    </test>
</suite>

The package testngtests contains classes MyTest1, MyTest2 with several test methods, all of which are empty:

@Test
public void testEins(){

}

etc.

Two of the tests use “dependsOnMethods”:

@Test(dependsOnMethods ="testEins")
public void testZwei(){}

@Test(dependsOnMethods ="testZwei")
public void testDrei(){}

If I run the suite using TestNG V 5.10-jdk15 or V 6.0.1, they always run in the same order:

testngtests.MyTest1.testEins()
testngtests.MyTest1.testZwei()
testngtests.MyTest1.testDrei() 
testngtests.MyTest2.testEinsA2()
testngtests.MyTest2.testDeins2()
testngtests.MyTest2.testDreiB2() 
testngtests.MyTest2.testZweiZ2()

as expected.

However, with TestNG 6.2.1, the order is:

MyTest1.testEins()
MyTest2.testDeins2()
MyTest2.testDreiB2()
MyTest2.testEinsA2()
MyTest2.testZweiZ2()
MyTest1.testZwei()
MyTest1.testDrei()

Note that now the tests from different classes are interleaved.

I expected tests from the same class to always run together (though not always in the same order). From the TestNG manual: “TestNG will try to group test methods by class.”

So did some change to TestNG break ordering in 6.2.1?

Or does TestNG simply not guarantee that test will be grouped by class? In that case the docs should be clarified to explain when TestNG will group by class - or not.

Also, in that case it would be nice to have a setting that forces grouping tests by class. Should I file a feature request for that?

[michabu]

Now, this really took me a long time to get back to. Thank you nevertheless very much for your help.

I think my main problem was to understand how to apply group-by-instance which is frankly quite embarrassing. But here you go:

This is how my setup looks like now in order to get the expected behaviour:

build.gradle:

test {
    useTestNG() {
        suites 'src/test/resources/testng.xml'
    }
}

dependencies {
    testCompile group: 'org.testng', name: 'testng', version: '6.14.3'
}

src/test/resources/testng.xml:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="all-tests" verbose="1">
    <test name="unit-tests" parallel="methods" preserve-order="false">
        <groups>
            <run>
                <include name="unit-test"/>
            </run>
        </groups>

        <packages>
            <package name="com.my.test.package.*"/>
        </packages>
    </test>

    <test name="integration-tests" parallel="false" group-by-instances="true">
        <groups>
            <run>
                <include name="integration-test"/>
            </run>
        </groups>

        <packages>
            <package name="com.my.test.package.*"/>
        </packages>
    </test>
</suite>

I hope this might help someone else who happens to stumble over the same issue. Thank you for making testNG work!

[saberduck]

I solved this by writing IAnnotationTransformer listener which sets distinct priority to every test class. However I still think that this is testNG issue and by default tests between different classes should not be interleaved, as it goes against intuitiveness

@Override
  public void transform(ITestAnnotation annotation, Class testClass, Constructor testConstructor, Method testMethod) {
    Class<?> declaringClass = testMethod.getDeclaringClass();
    Integer priority = priorityMap.get(declaringClass);
    if (priority == null) {
      priority = priorityCounter++;
      priorityMap.put(declaringClass, priority);
    }
    log.trace("Setting priority {} for {}", priority, testMethod);
    annotation.setPriority(priority);
  }