Using softAssert and "hard" Asserts together, softAsserts are swallowed by the "hard" Asserts

TestNG Version

7.1.0, latest from the maven repo https://mvnrepository.com/artifact/org.testng/testng

Issue

This issue is (probably) not one with testng per se but rather an architectural problem I have using testng and softAssertions. I found a solution and my questions are if there is a better solution to this and if there is a reason why I should not use my solution to this problem.

Essentially what I am doing is this:

softAssert.fail("SoftAssert fail");
assert.fail("assert fail");
softAssert.assertAll();

What happens is that assert.fail("assert fail"); obviously fails and the exception is shown as java.lang.AssertionError: assert fail but the softAssertion is never evaluated because of the Exception. The softAssertions will never be shown until the normal assert does not fail. But I want the failed softAsserts to be shown as well.

So what I ended up doing is this: Simply extend SoftAssert so I can use the “doAssert” method, because it is protected in the original class.

public class MySoftAssert extends SoftAssert {
    @Override
    protected void doAssert(IAssert<?> a) {
        super.doAssert(a);
    }
}

Then define a HardAssert class, which also passes all asserts to ‘the’ softAssert object used by the tests to do assertions, and when the hardAssert fails it automatically evaluates all the softAsserts, which also includes the just done hardAssertion.

public class MyHardAssert extends SoftAssert {
    private final MySoftAssert softAssert;

    public MyHardAssert(MySoftAssert softAssert){
        this.softAssert = softAssert;
    }

    @Override
    public void onAssertFailure(IAssert<?> assertCommand, AssertionError ex) {
        super.onAssertFailure(assertCommand, ex);
        softAssert.assertAll("Hard Assert failed, also showing all soft asserts that failed before:");
    }

    @Override
    protected void doAssert(IAssert<?> a) {
        softAssert.doAssert(a);
        super.doAssert(a);
    }
}

Now I can do this:

softAssert = new MySoftAssert();
hardAssert = new MyHardAssert(softAssert);
softAssert.fail("softAssert fail");
hardAssert.fail("hardAssert fail");
...

and it will show both asserts as being failed.

java.lang.AssertionError: The following asserts failed:
	softAssert fail
	hardAssert fail

What I am also using is a test wrapper/listener that automatically evaluates all softAsserts at the end, but it is not called when the tests fail because of a ‘normal’ assertion. That wrapper and the hardAssert is inspired by code I found here: https://github.com/cbeust/testng/issues/1038#issuecomment-217466619

public class SoftAssertListener implements IHookable {

    @Override
    public void run(IHookCallBack iHookCallBack, ITestResult iTestResult) {
        iHookCallBack.runTestMethod(iTestResult);
        Class<?> testClass = iTestResult.getTestClass().getRealClass();
        Field field = null;
        SoftAssert softAssert = null;
        try {
            field = testClass.getField("softAssert");
            field.setAccessible(true);
            softAssert = (SoftAssert) field.get(iTestResult.getInstance());
        } catch (NoSuchFieldException | IllegalAccessException e) {
            e.printStackTrace();
        }

        if(softAssert != null) {
            softAssert.assertAll();
        }
    }
}

Is the issue reproductible on runner?

• IntelliJ

[krmahadevan]

@wulfihm -

If I use softasserts and normal asserts, the softasserts are only evaluated if the asserts pass.

This is NOT true. Please go back to my implementation and pay closer

    callBack.runTestMethod(testResult);
    try {
      softAssert.assertAll();
    } catch (AssertionError e) {
      throw new AssertionError(e.getMessage(), testResult.getThrowable());
    }

The above implementation would invoke softAssert.assertAll() irrespective of whether Assert.xxx causes a pass (or) a fail.

Since we have reached a conclusion that what I proposed is not something that you want, but you want to mix both Soft and Hard asserts together in a manner that I fail to understand, I will step aside and let someone else chime in.

The fact that you are doing something like this

protected void doAssert(IAssert<?> a) {
        softAssert.doAssert(a);
        super.doAssert(a);
    }

in your customised MyHardAssert is already caused the hard assert to lose its value. Because this is just soft assertion.

You can test it out by flipping the order

hardAssert.fail("hardAssert fail");
softAssert.fail("softAssert fail");

If your code still reaches softAssert.fail() then its just soft assertion that you are doing.

@juherr - Your thoughts ?

[krmahadevan]

That is the whole point why I implemented the approach to evaluate all soft asserts if a hard assert fails.

You are diluting the concept of Hard assert, if you want all the test to continue to evaluate all the soft asserts even after a hard assert has failed.

If you build something like this, this would ensure that the soft assertion would get called if and only if all hard asserts pass. If the hard assert fails, even then the soft assert would get called.

import org.testng.IHookCallBack;
import org.testng.IHookable;
import org.testng.ITestResult;
import org.testng.Reporter;
import org.testng.asserts.SoftAssert;

public class BaseTestCase implements IHookable {

  private static final String KEY = "softassert";

  @Override
  public void run(IHookCallBack callBack, ITestResult testResult) {
    SoftAssert softAssert = new SoftAssert();
    testResult.setAttribute(KEY, softAssert);
    callBack.runTestMethod(testResult);
    try {
      softAssert.assertAll();
    } catch (AssertionError e) {
      throw new AssertionError(e.getMessage(), testResult.getThrowable());
    }
  }

  protected final SoftAssert getSoftAssertForCurrentTest() {
    Object object = Reporter.getCurrentTestResult().getAttribute(KEY);
    if (object == null) {
      throw new RuntimeException("no soft assert object found");
    }
    if (!(object instanceof SoftAssert)) {
      throw new IllegalStateException("Not a soft assertion object");
    }
    return (SoftAssert) object;

  }
}

Test class looks like below

import org.testng.Assert;
import org.testng.annotations.Test;

public class SimpleTestCase extends BaseTestCase {

  @Test
  public void testMethod() {
    getSoftAssertForCurrentTest().assertTrue(false, "Soft assert failed");
    Assert.assertFalse(true, "Hard assert failed");
    getSoftAssertForCurrentTest().assertTrue(true);
  }

}