question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

JUnite reporter xml results are not within JUnit reporting standards

See original GitHub issue
  1. Newman Version (can be found via newman -v): 3.8.3
  2. OS details (type, version, and architecture): using Docker image: postman/newman_ubuntu1404
  3. Are you using Newman as a library, or via the CLI? CLI in Docker postman/newman_ubuntu1404
  4. Did you encounter this recently, or has this bug always been there: I’m new to the tool
  5. Expected behaviour: JUnit reporter creates properly formatted XML file
  6. Command / script used to run Newman: newman run "https://api..." --environment="https://api..." -x --reporters junit --reporter-junit-export ./test-reports/bunker-api-test-results.xml
  7. Sample collection, and auxiliary files (minus the sensitive details): super simple collection with one endpoint and 3 tests. I have one test intentionally failing.
  8. Screenshots (if applicable): Below is the resulting XML file generated.
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="bunker-service-api" time="0.963">
  <testsuite name="BusinessEntityType / BusinessEntityType" id="6c63aa1e-1fb2-4146-b555-0ecdf25d0d15" tests="3" time="0.963">
    <testcase name="BusinessEntityType has required fields" time="0.963"/>
    <testcase name="6 entity types are returned" time="0.963"/>
    <testcase name="the tests are now failing" time="0.963">
      <failure type="AssertionFailure">
        <![CDATA[Failed 1 times.]]>
      </failure>
    </testcase>
  </testsuite>
</testsuites>

The above XML file is what is generated. I am working with Bitbucket Pipelines which should pick-up the JUnit xml file and report on it. However, pipelines out says:

Located test suite containing 3 tests, with 0 failures and 0 errors.
Finished scanning for test reports. Found 1 test report files.
Merged test suites, total number tests is 3, with 0 failures and 0 errors.
Uploading test results
Finished uploading test results.

Bitbucket support says that the XML generated should look more like the following in order to be compliant with http://help.catchsoftware.com/display/ET/JUnit+Format

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="bunker-service-api" time="0.963">
  <testsuite name="BusinessEntityType / BusinessEntityType" id="6c63aa1e-1fb2-4146-b555-0ecdf25d0d15" tests="3" failures="1" errors="0" time="0.963">
    <testcase name="BusinessEntityType has required fields" time="0.963"/>
    <testcase name="6 entity types are returned" time="0.963"/>
    <testcase classname="JUnitXmlReporter.constructor" name="the tests are now failing" time="0.963">
      <failure type="AssertionFailure">
        <![CDATA[Failed 1 times.]]>
      </failure>
    </testcase>
   </testsuite>
</testsuites>

Basically, the testsuite and testcase should look more like this. <testsuite name="BusinessEntityType / BusinessEntityType" id="6c63aa1e-1fb2-4146-b555-0ecdf25d0d15" tests="3" failures="1" errors="0" time="0.963"> <testcase classname="JUnitXmlReporter.constructor" name="the tests are now failing" time="0.963">

Issue Analytics

  • State:closed
  • Created 6 years ago
  • Comments:6 (2 by maintainers)

github_iconTop GitHub Comments

1reaction
atqmetrycommented, Nov 13, 2018

Not sure whether I should create a separate issue for this, but only failed test cases have classname in XML. I’ve just verified this with newman 4.2.2 on Windows 10 64-Bit.

AFAIK JUnit XML format has classname in all test cases irrespective of their status. We are having trouble with such implementation. Is it possible to add classname attribute for all test case elements in nearest future or this is intentionally created so?

Please correct me if I’m missing anything here. Looking forward to your response.

0reactions
BrambleRamblecommented, Jul 5, 2018

@kunagpal That’s brilliant, thanks for turning it around so quickly too!

Read more comments on GitHub >

github_iconTop Results From Across the Web

JUnit report is not showing my tests (#223132) - GitLab.org
In pipelines, we see the new tests tab but it simply says "There are no tests to show." We can see that the...
Read more >
Chapter 7. Reporting JUnit results - liveBook · Manning
This chapter covers various tools and techniques available for reporting JUnit test results, including extending JUnit to write your own custom reporting ......
Read more >
JUnit XML Reporter | Developer Guide - Nightwatch.js
JUnit is a widespread XML format for generating test results and most CI system (e.g. Jenkins) have native support to plug this in...
Read more >
What is the JUnit XML format specification that Hudson ...
I have Hudson as continuous integration server and I want to use option 'Publish JUnit test result report'. But ...
Read more >
Taking advantage of JUnit XML reports - Xray Documentation
JUnit was massively used by the Java community and thus, its XML test result reports have become a de facto standard for test...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found