Feature: allow modification of group array via IInvokedMethod

TestNG Version

Note: only the latest version is supported

Feature Request

Adding the ability to modify/set groups from IInvokedMethod, via the IInvokedMethodListener.

If I’m not mistaken too, using alwaysRun = true in conjunction with dependsOnMethods does not work as expected, so using the groups tags here would also help resolve that bug as a workaround, although it’s not the primary reason for this feature request.

Actual behavior

We’re not able to modify groups at runtime via this listener or via IInvokedMethod (which could be for a valid reason).

I’m not sure if there’s an underlying reason for this, perhaps by the time you’d be able to modify the groups, the annotated methods themselves will already have been set in stone. If that’s the case I think this would be a bigger feature request.

Use case sample

Please, share the test case (as small as possible) which shows the issue

Let’s say you’d like every @Test that’s in the project to be assigned to a default group, say ‘regression’. Instead of going into hundreds of classes and adding that ‘regression’ group each time (and because we’re human, missing a few here and there) like we do now, you could use the IInvokeMethodListener and do the following:

public class AddDefaultGroupListener implements IInvokedMethodListener {

  @Override
  public void beforeInvocation(IInvokedMethod iInvokedMethod, ITestResult iTestResult) {
    if (iInvokedMethod.isTestMethod()){
        iInvokedMethod.setGroups("defaultGroupName");
        // or optionally, adding to existing groups
        iInvokedMethod.addGroup("defaultGroupName");
    }
  }

Of course you could also be more specific to some situations in your project for example, where perhaps certain @BeforeMethods named a certain way should belong to a group and you’d like to just code that in one place.

In my use case, it could also be used to mark all @BeforeMethods with specific groups that will always run specific @BeforeMethods, without needing to add in the alwaysRun = true to every @BeforeMethod like we do now.

public class AddGroupsToEveryBeforeMethod implements IInvokedMethodListener {

  @Override
  public void beforeInvocation(IInvokedMethod iInvokedMethod, ITestResult iTestResult) {
    if (iInvokedMethod.isConfigurationMethod() && iInvokedMethod.getTestMethod()
        .isBeforeMethodConfiguration()) {
      iInvokedMethod.getTestMethod().setGroups("defaultGroupName");
      // also optionally, adding to existing groups
      iInvokedMethod.addGroup("defaultGroupName");
    }
}

I think a feature like this would add even more customization from the IInvokedMethodListener and could open up for some pretty slick dynamic group tagging. It would allow for those who cared to use it, the ability to remove a portion of the human error that can occur during test group tagging in a large scale project.

I’m sure we’ve all wondered where that one @Test was that you made and realized that you forgot to mark it with a specific group that you run each night 😉

[krmahadevan]

@Proryanator - I think you are basically getting the concept backwards.

Let me explain some of the facts and maybe that will add clarity

• Groups in testng is basically a selection mechanism using which you tell TestNG what to execute and what to skip.
• Since its a selection mechanism, it gets applied much ahead in the life cycle so that TestNG can determine what would need to be executed.
• IAnnotationTransformer listener is a callback listener that TestNG provides to you to give you an opportunity to change some of the attributes of either a @Test or a @Before/@After configuration and it gets applied while TestNG is determining what to run via the filtering mechanism (The other filtering mechanisms via a suite xml would include, package names or just class names or just method names within a class name etc.,)
• Once TestNG has determined what to run, there’s no way you can alter it (which means except for IAnnotationTransformer interface, you dont get to change what can be run and what can|should be skipped. The other filtering mechanisms are ofcourse method selectors (either beanshell or org.testng.IMethodSelector or org.testng.IMethodInterceptor)
• Not all configuration methods are agnostic of the test class instance. For example @BeforeClass, @AfterClass, @BeforeMethod and @AfterMethod are very much bound to a test class instance)
• IAnnotationTransformer runs once per @Test method and once per every configuration method because here TestNG is just giving you what all methods are eligible for execution. At this point in time the context behind these methods are not applicable (when I say context, what I mean is how to handle a particular test method. for e.g, a test method could be bound to a data provider or may have an invocationcount which means the same test method would need to called multiple times. But from an annotation perspective which is what an IAnnotationTransformer is all about, its just 1 single method. And the same goes for configuration methods as well)
• Last but not the least, the whole intent of providing an IAnnotationTransformer is so that you the user can alter the attributes of an annotation. Annotations in Java are static and not dynamic, which means that once the JVM instantiates and assigns values to an annotation, it cannot be changed|altered easily. There are hacks which you can employ to do it, but they are very specific to JDK versions and perhaps even JDK flavors. For e.g., I wrote about once such approach in my blog post. You can read more about it here

Hope that adds clarity

[juherr]

Closed because of the need behind the feature request can be solved in another way.