JUnite reporter xml results are not within JUnit reporting standards
See original GitHub issue- Newman Version (can be found via
newman -v
):3.8.3
- OS details (type, version, and architecture): using Docker image:
postman/newman_ubuntu1404
- Are you using Newman as a library, or via the CLI? CLI in Docker
postman/newman_ubuntu1404
- Did you encounter this recently, or has this bug always been there: I’m new to the tool
- Expected behaviour: JUnit reporter creates properly formatted XML file
- Command / script used to run Newman:
newman run "https://api..." --environment="https://api..." -x --reporters junit --reporter-junit-export ./test-reports/bunker-api-test-results.xml
- Sample collection, and auxiliary files (minus the sensitive details): super simple collection with one endpoint and 3 tests. I have one test intentionally failing.
- Screenshots (if applicable): Below is the resulting XML file generated.
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="bunker-service-api" time="0.963">
<testsuite name="BusinessEntityType / BusinessEntityType" id="6c63aa1e-1fb2-4146-b555-0ecdf25d0d15" tests="3" time="0.963">
<testcase name="BusinessEntityType has required fields" time="0.963"/>
<testcase name="6 entity types are returned" time="0.963"/>
<testcase name="the tests are now failing" time="0.963">
<failure type="AssertionFailure">
<![CDATA[Failed 1 times.]]>
</failure>
</testcase>
</testsuite>
</testsuites>
The above XML file is what is generated. I am working with Bitbucket Pipelines which should pick-up the JUnit xml file and report on it. However, pipelines out says:
Located test suite containing 3 tests, with 0 failures and 0 errors.
Finished scanning for test reports. Found 1 test report files.
Merged test suites, total number tests is 3, with 0 failures and 0 errors.
Uploading test results
Finished uploading test results.
Bitbucket support says that the XML generated should look more like the following in order to be compliant with http://help.catchsoftware.com/display/ET/JUnit+Format
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="bunker-service-api" time="0.963">
<testsuite name="BusinessEntityType / BusinessEntityType" id="6c63aa1e-1fb2-4146-b555-0ecdf25d0d15" tests="3" failures="1" errors="0" time="0.963">
<testcase name="BusinessEntityType has required fields" time="0.963"/>
<testcase name="6 entity types are returned" time="0.963"/>
<testcase classname="JUnitXmlReporter.constructor" name="the tests are now failing" time="0.963">
<failure type="AssertionFailure">
<![CDATA[Failed 1 times.]]>
</failure>
</testcase>
</testsuite>
</testsuites>
Basically, the testsuite and testcase should look more like this.
<testsuite name="BusinessEntityType / BusinessEntityType" id="6c63aa1e-1fb2-4146-b555-0ecdf25d0d15" tests="3" failures="1" errors="0" time="0.963">
<testcase classname="JUnitXmlReporter.constructor" name="the tests are now failing" time="0.963">
Issue Analytics
- State:
- Created 6 years ago
- Comments:6 (2 by maintainers)
Top Results From Across the Web
JUnit report is not showing my tests (#223132) - GitLab.org
In pipelines, we see the new tests tab but it simply says "There are no tests to show." We can see that the...
Read more >Chapter 7. Reporting JUnit results - liveBook · Manning
This chapter covers various tools and techniques available for reporting JUnit test results, including extending JUnit to write your own custom reporting ......
Read more >JUnit XML Reporter | Developer Guide - Nightwatch.js
JUnit is a widespread XML format for generating test results and most CI system (e.g. Jenkins) have native support to plug this in...
Read more >What is the JUnit XML format specification that Hudson ...
I have Hudson as continuous integration server and I want to use option 'Publish JUnit test result report'. But ...
Read more >Taking advantage of JUnit XML reports - Xray Documentation
JUnit was massively used by the Java community and thus, its XML test result reports have become a de facto standard for test...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Not sure whether I should create a separate issue for this, but only failed test cases have classname in XML. I’ve just verified this with newman 4.2.2 on Windows 10 64-Bit.
AFAIK JUnit XML format has classname in all test cases irrespective of their status. We are having trouble with such implementation. Is it possible to add classname attribute for all test case elements in nearest future or this is intentionally created so?
Please correct me if I’m missing anything here. Looking forward to your response.
@kunagpal That’s brilliant, thanks for turning it around so quickly too!