@BeforeMethods: alwaysRun=true used with dependsOnMethods does not work as it should

TestNG Version

Note: only the latest version is supported

Expected behavior

When using both alwaysRun=true and dependsOnMethods, you would expect a @BeforeMethod to run no matter the group it belongs to (direct quote from testng’s docs: https://testng.org/doc/documentation-main.html), and you’d expect if any methods defined in dependsOnMethods failed, this @BeforeMethod would not run.

Actual behavior

I’ve observed that indeed alwaysRun=true does make this method run no matter the group, however alwaysRun=true seems to take precedence over dependsOnMethods, causing it to be ignored. This means that when working with multiple groups, and using alwaysRun=true, you can’t setup config method dependencies and you’ll always have your @BeforeMethods run no matter what.

Is the issue reproduce-able on runner?

• Shell
• Maven
• IntelliJ

Test case sample

I’ve attached the code I’m specifying here: issue_2135.zip

We have a top parent class, which defined a @BeforeMethod that should always run. We force this method to do a hard assertion to always make this @BeforeMethod considered failed.

public class GrandpaBearGitHubTest {
  @BeforeMethod(alwaysRun = true)
  public void mobileDriverCreation() {
    Assert.fail("Could not create the driver at all...");
  }
}

The first class below this one, with the alwaysRun=true and dependsOnMethods, should not run if the @BeforeMethod called GrandpaBearGitHubTest.mobileDriverCreation() fails:

public class PapaBearGithubTest extends GrandpaBearGitHubTest {

  @BeforeMethod(alwaysRun = true, dependsOnMethods = {"mobileDriverCreation"})
  public void makeMostPages() {
    System.out.println("Ran makeMostPages");
  }
}

Here’s the simple @Test class:

public class GrandsonBearGithubTest extends PapaBearGithubTest {

  @Test(groups = "yogi-bear")
  public void testMethod() {
    System.err.println("Running test method");
  }
}

Here’s the results of running both with/without -Dgroups=yogi-bear. Both times the child class @BeforeMethod PapaBearGithubTest.makeMostPages() still runs (as you can see when it prints ‘Ran makeMostPages’, even though the method it depends on has failed:

[Run with -Dgroups] [Run without -Dgroups]

I also ran it in IntelliJ to be sure it’s about TestNG and not about the way that maven is handling groups, the results were the same:

An even further exploration/test, I added a second @BeforeMethod to the PapaBearGithubTest to use the specific group itself instead of alwaysRun=true and that @BeforeMethod did not run, indicating that the behavior issue revolves around alwaysRun=true alone and not groups:

public class PapaBearGithubTest extends GrandpaBearGitHubTest {

  @BeforeMethod(alwaysRun = true, dependsOnMethods = {"mobileDriverCreation"})
  public void makeMostPages() {
    System.out.println("Ran makeMostPages");
  }

  @BeforeMethod(groups = {"yogi-bear"}, dependsOnMethods = {"mobileDriverCreation"})
  public void makeOtherPages() {
    System.out.println("Ran makeOtherPages");
  }
}

If the new PapaBearGithubTest.makeOtherPages() would have run we would see ‘Ran makeOtherPages’ but we see no such output, just the ‘Ran makeMostPages’ from the first one, so we know that using groups and dependsOnMethods works as expected:

[juherr]

@Proryanator Feel free to update the doc which is located here: https://github.com/testng-team/testng-team.github.io

BTW, you can try to fix the current issue too and we will happy to help you 👍

[Proryanator]

I am still interested in working on this but, I’m mostly working on this as a support to our testing efforts for my job 😄 when work settles down a bit, I’ll jump back into making progress on this 💪🏻