Test data as code feature concept

TestNG Version

Note: only the latest version is supported 7.5.0

The point of this concept is to write test data files as code on kotlin (or any other language). This allows to put dynamic data (i.e. generated on each launch, for example, today’s date or random string) into data files.

The problem:

I was involved in a project, where test data was stored in yaml files. They parsed those yaml files, and used it in a tests. The problem was, they needed dynamic values, like today’s date, or random sequence of characters. This data cannot be places in yaml testdata files by default. So they used custom markup language, to define, for example random alphabetic sequence of length 6. This custom markup language was processed by custom yaml deserializer, to replace markup with calculated values. In this case, why can’t we write test data in code?

How test data as kotlin code may look like? Kotlin code may seem like declarative language, when using named constuctors for example:

data class Person(
    val name: String,
    val age: Int,
    val profession: String
)

val person = Person(
    name = "Cris",
    age = 18,
    profession = "student"
) 

The problem with this approach is that all data-generation code, and consequently data(because it’s essentially hardcoded) remains in RAM, and may lead to OOM. We may overcome this with custom classloader, for example:

data class Person(
    val name: String,
    val age: Int,
    val profession: String
)

val person = getPerson()

fun getPerson() {
    /**
     * This custom classloader is used to load code that provides data
     * When loaded code executed, and method returned, classloader, and all
     * classes, loaded by it, will be cleared by GC.
     * */
    val dataClassLoader = DataClassLoader()
    val providerClass = dataClassLoader.loadClass("PersonProvider")
    val providerMethod = providerClass.declaredMethod("provide")
    val person = providerMethod.invoke(
        providerClass.getDeclaredConstructor().newInstance()
    )

    return person
}


class PersonProvider {
    fun provide() = Person(
        name = "Cris",
        age = 18,
        profession = "student"
    )
}

How a test class may look like?

class TestClass {
    @Test(dataAsCode = [["PersonProvider#provide"]])
    fun test(person: Person) {
        
    }
}

What do you guys think? Would you accept such feature in testNG?

[dsankouski]

@krmahadevan

Suppose, you’re decided to use data provider, like in example:

import org.testng.annotations.DataProvider;

class DataProviders {
    @DataProvider
    public static Object[][] dp() {
        return new Object[][] {
                {"Mike", 34, "student"},
                {"Mike", 23, "driver"},
                {"Paul", 20, "director"}
        };
    }
}

class Test {
    @Test(dataprovider = "dp", dataproviderClass = DataProviders.class)
    public void test(String name, int age, String prof) {
        // test code
    }
}

All data in DataProviders class will be placed in constant pool $4.4 upon compilation When loading DataProviders.class $5.3, it’s internal binary representation will be created in the method area $2.5.4. This binary representation contains per-class runtime constant pool $2.5.5,
which contains all the data we need for test. So our code will basically “copy” data from runtime constant pool to Object[][] instance: $4.4: Java Virtual Machine instructions do not rely on the run-time layout of classes, interfaces, class instances, or arrays. Instead, instructions refer to symbolic information in the constant_pool table

When dp method is invoked, Object[][] instance is created in heap area, thus duplicating data, contained in method area.

With current DP implementation we end up with data being duplicated in heap and in method area

Since we only need data itself, i.e. Object[][] instance, we may load DataProviders class on demand with custom classloader. It will be garbage collected, along with all loaded classes, when there’ll be no links.

Few words about interned strings: Though there’s string literal pool in java, it’s located in heap, and represent string objects. String literals in class contant pool are also “interned” in a sense, that unique literal contained only once in a class file, and bytecode instructions like LDC "Mike" refer to string index in constant pool. See also $4.4.3 $4.4.7

Note, $2.5.4 states: Although the method area is logically part of the heap, simple implementations may choose not to either garbage collect or compact it. This specification does not mandate the location of the method area or the policies used to manage compiled code This means, that method area garbage collecting is implementation dependent.

[krmahadevan]

@dsankouski

So, what I’m thinking of, is to add dataProviderClass string parameter to @Test annotation, and if that’s not empty, load dataprovider via custom classloader, when resolving it’s data.

Yes. It has to be a string parameter so that we can use reflection backed by a custom class loader to be used to load up the class. Then again, I think you would also need to consider providing a means to inject a custom class loader itself. So would it basically mean that we would now need to add up 2 string parameters (1 which specifies the data provider class and the other which specifies the custom class loader that we would like to be used.). I say this because if we end up specifying the classloader as a class parameter then we are back to square one.