ITestResult Thread Not Used in TestListenerAdapter Extended Classes onTestSuccess/Failure/Skipped

Summary

When configuring a setupMethod with ITestResult, annotated with @BeforeMethod, and then populating attributes to be used in correlating the failure in a test method (@Test), the ITestResult thread is not available to the listener class extending the TestListenerAdapter.

Note, this works in version 6.14.0, but fails starting 7.0.0.

TestNG Version

7.3.0

Expected behavior

ITestResult Thread to be populated with attributes assigned in a test method.

Actual behavior

Only the ITestResults populated prior to the @Test method are available onTestSuccess (or onTestFailure or onTestSkipped)

Is the issue reproducible on runner?

• Shell
• Maven
• Gradle
• Ant
• Eclipse
• IntelliJ
• NetBeans

Test case sample

package org.mygroup;

import org.testng.ITestResult;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;

import static org.testng.Assert.assertFalse;

public class TestSample {

  private ThreadLocal<ITestResult> tr = new ThreadLocal<>();

  TestSample() {
  }

  @BeforeMethod(alwaysRun = true)
  public void setupMethod(ITestResult tr) {
    this.tr.set(tr);
    tr.setAttribute("field1", "value1");
    tr.setAttribute("field2", "value2");
    System.out.println("Attribute names size in Setup: " + tr.getAttributeNames().size());

  }

  @AfterClass(alwaysRun = true)
  public void tearDown() {
    tr.remove();
  }

  @Test(groups = {"acceptance"})
  public void testInvocation() {
    //These values aren't included in the Listener after v7.0.0
    tr.get().setAttribute("field3", "value3");
    tr.get().setAttribute("field4", "value4");
    System.out.println("Attribute names size in test: " + tr.getAttributeNames().size());

    assertFalse(true);
  }
}

The Listener that reveals only the first two fields are loaded:

package org.mygroup;

import java.util.HashMap;
import java.util.Map;
import java.util.Set;

import org.testng.ITestResult;
import org.testng.TestListenerAdapter;

public class Listener extends TestListenerAdapter {

  @Override
  public void onTestFailure(ITestResult tr) {
    getTestResultLogObject(tr);
  }

  @Override
  public void onTestSkipped(ITestResult tr) {
    printTestResultAttributes(tr);
  }

  @Override
  public void onTestSuccess(ITestResult tr) {
    getTestResultLogObject(tr);
  }

  void printTestResultAttributes(ITestResult tr) {
    System.out.println("Attribute names size in Listener: " + tr.getAttributeNames().size());
    System.out.println(getAttributes(tr));
  }

  private Map<String, String> getAttributes(ITestResult tr) {
    Set<String> names = tr.getAttributeNames();
    Map<String, String> attributes = new HashMap<>();
    for (String name : names) {
      Object value = tr.getAttribute(name);
      if (null == value) {
        attributes.put(name, "null");
      }
      else {
        attributes.put(name, value.toString());
      }
    }
    return attributes;
  }

}

[krmahadevan]

@egglestonbd - Please also note that, TestNG by default would ensure that even a beforeInvocation() and afterInvocation() listener call happens on the same thread. Here’s a cleaned up version of your same test code (This is using TestNG 7.3.0)

import org.testng.ITestResult;
import org.testng.Reporter;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

import static org.testng.Assert.fail;

@Listeners(MySampleListener.class)
public class TestSample {

  @BeforeMethod(alwaysRun = true)
  public void setupMethod(ITestResult tr) {
    tr.setAttribute("field1", "value1");
    tr.setAttribute("field2", "value2");
    System.out.println("Attribute names size in Setup: " + tr.getAttributeNames().size());
  }

  @Test
  public void testInvocation() {
    //This gets hold of the test result of the current test method.
    ITestResult testResult = Reporter.getCurrentTestResult();
    testResult.setAttribute("field3", "value3");
    testResult.setAttribute("field4", "value4");
    System.out.println("Attribute names size in test: " + testResult.getAttributeNames().size());
    fail("Simulating a failure");
  }
}

import java.util.Map;
import java.util.Optional;
import java.util.stream.Collectors;
import org.testng.IInvokedMethod;
import org.testng.IInvokedMethodListener;
import org.testng.ITestResult;

public class MySampleListener implements IInvokedMethodListener {

  @Override
  public void afterInvocation(IInvokedMethod method, ITestResult testResult) {
    printTestResultAttributes(testResult);
  }

  void printTestResultAttributes(ITestResult tr) {
    String msg = String.format("[%s] method [%s] had [%d] attributes and its status was %s",
        parseMethodType(tr),
        tr.getMethod().getQualifiedName(),
        tr.getAttributeNames().size(),
        parseStatus(tr)
    );
    System.out.println(msg);
    System.out.println(getAttributes(tr));
  }

  private static String parseMethodType(ITestResult tr) {
    return (tr.getMethod().isTest() ? "test" : "configuration");
  }

  private static String parseStatus(ITestResult tr) {
    switch (tr.getStatus()) {
      case ITestResult.SUCCESS:
      case ITestResult.SUCCESS_PERCENTAGE_FAILURE:
        return "passed";
      case ITestResult.FAILURE:
        return "failed";
      case ITestResult.SKIP:
        return "skipped";
      default:
        return "unknown";
    }
  }

  private Map<String, String> getAttributes(ITestResult tr) {
    return tr.getAttributeNames()
        .stream()
        .collect(
            Collectors.toMap(
                name -> name,
                name -> Optional.ofNullable(tr.getAttribute(name))
                    .orElse("null").toString()
            )
        );
  }
}

And here’s the output

Attribute names size in Setup: 2
[configuration] method [com.rationaleemotions.github.issue2478.TestSample.setupMethod] had [0] attributes and its status was passed
{}
Attribute names size in test: 4
[test] method [com.rationaleemotions.github.issue2478.TestSample.testInvocation] had [4] attributes and its status was failed
{field1=value1, field3=value3, field2=value2, field4=value4}

java.lang.AssertionError: Simulating a failure

	at org.testng.Assert.fail(Assert.java:99)
	at com.rationaleemotions.github.issue2478.TestSample.testInvocation(TestSample.java:28)

Closing this issue as requested.

[juherr]

Hi,

Could you try without the ThreadLocal and passing the ITestResult as test param? testInvocation(ITestResult tr)?