[Bug] JUnit Reporter doesn't include ignored methods

TestNG Version

6.9.13.6 (latest available)

Expected behavior

I believe that JUnitReportReporter should be compliant with the JUnit 4.12 rules and thus should include the disabled/ ignored tests (the ones marked with @Test(enabled = false) in the XML report under junitreports/ directory.

Yes, I know, I know, this is not how TestNG treats ignored tests - but if we are going to produce a JUnit-compliant test report, then I guess we should stick to the JUnit rules 😃

Actual behavior

Now, TestNG doesn’t generate absolutely no info about such test in the output xml report.

Is the issue reproductible on runner?

Tested via Maven.

Test case sample

 @Test(enabled = false)
 public void should_be_ignored() {
     assertEquals("abcd", "abc");
 }

I would expect to see followig entry in the generated JUnit report:

<testcase time="0" classname="com.myproject.AbcTest" name="should_be_ignored">
  <skipped/>
</testcase>

[gdemecki]

@krmahadevan sure, let me explain:

I don’t see the attribute called “ignored” in the JUnit schema you shared. But now TestNG added it up via #1247

It is Ok that you cannot see the attribute ignored there. Because JUnit-style of report doesn’t use ignored word, it just uses the skipped for this purpose. In the JUnit world they are synonyms. Although in the TestNG world they aren’t synonyms.

How will an end user who looks at the junitreports/xml file be able to differentiate which test was skipped and which one was ignored ?

Please look at my explanation above. They are synonyms.

I think your confusion is caused by differences in the JUnit and TestNG possible test statues. Basically:

JUnit has four possible test statuses:

1. Success
2. Failure (due to AssertionFailedError)
3. Error (unexpected Exception has been thrown)
4. Skipped (aka. Ignored)

While TestNG has so far just three (please correct me if I’m wrong) possible test statuses:

1. Passed
2. Failed
3. Skipped (but not ignored! This status is only for tests which were dependent on the failed tests)

So TestNG has no equivalent for the status SKIPPED from JUnit (although it has already status with the same name, but for a totally different purpose). This is probably root of a confusion.

On the other hand: JUnit has never had any notion of dependency between tests, so there is no equivalent of the TestNG SKIPPED status.

So thanks to your work there is a new status in the TestNG which is called IGNORED - which will be/ should be an equivalent for SKIPPED status from junit. Because so far, end user was totally not aware of how many tests were actually permanently disabled in a test suite. Now it will change.

Thanks, and please ping me if something is not clear.

Ps. @cbeust , @juherr please correct me if I was wrong somewhere above.

[krmahadevan]

@juherr Thank you. Can you please help approve the PR and kindly help it get merged ?