Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

When group-by-instances is set to true the instances created by @Factory does not run in parallel

See original GitHub issue

Running this on TestNG 6.8:

package test;

import org.apache.log4j.Logger;
import org.testng.annotations.*;
import org.testng.annotations.Test;

public class TestTest
{
    private String param;

    @Factory( dataProvider = "prov" )
    public TestTest( String param )
    {
        this.param = param;
    }

    @DataProvider( name = "prov" )
    public static Object[][] dataProvider()
    {
        System.out.println( "[" + Thread.currentThread().getId() +  "] Provide data" );
        return new Object[][] {
                { "One" },
                { "Two" },
                { "Three" },
        };
    }

    @BeforeClass
    public void prepare()
    {
        System.out.println( "[" + Thread.currentThread().getId() +  "] Prepare " + param );
    }

    @Test
    public void test1()
    {
        System.out.println( "[" + Thread.currentThread().getId() +  "] Test1 " + param );
    }

    @Test( dependsOnMethods = "test1" )
    public void test2()
    {
        System.out.println( "[" + Thread.currentThread().getId() +  "] Test2 " + param );
        sleep();
    }

    @AfterClass
    public void clean()
    {
        System.out.println( "[" + Thread.currentThread().getId() +  "] Clean " + param );
    }

    private void sleep() {
        try {
            Thread.sleep(10000);
        } catch (Exception ignored) {}
    }
}

With the following testng.xml:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >

<suite name="Suite" parallel="instances" thread-count="5" group-by-instances="true">
  <test name="Tests">
    <classes>
       <class name="test.TestTest" />
    </classes>
  </test>
</suite>

Results in:

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running TestSuite
[1] Provide data
[9] Prepare Three
[9] Test1 Three
[10] Test2 Three
[10] Clean Three
[11] Prepare Two
[11] Test1 Two
[12] Test2 Two
[12] Clean Two
[13] Prepare One
[13] Test1 One
[13] Test2 One
[13] Clean One
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.61 sec

With Thread.sleep(), I’m expecting the other threads to kick in and that the runtime should be just over 10 seconds. This is true when group-by-instances is set to false, but I need my test cases to run together for my WebDriver tests not to time out.

I’m also slightly confused over the thread-id’s. From the documentation: parallel="instances": TestNG will run all the methods in the same instance in the same thread, but two methods on two different instances will be running in different threads. The output above seems to indicate that this is not the case, but methods from the same instance is running on two different threads, even if they are executed in the right order.