Junit formatter is not outputting according to XSD Schema

Describe the bug When using the JUnit formatter (--format=junit --output=test.xml) then the output XML is not aligned with the XSD Schema (https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd).

JUnit only supports “error” or “failure”, not “warning”, “hint” etc.

To Reproduce

1. Given an OpenAPI document that doesn’t have an operationId should create a warning
2. Run this CLI command lint myopapi.yaml --format=junit --output=test.xml
3. See test.xml

<?xml version="1.0" encoding="utf-8"?>
<testsuites>
    <testsuite package="org.spectral" time="0" tests="2" errors="2" name="openapi.yaml">
        <testcase time="0" name="org.spectral.operation-description" classname="openapi">
            <warning message="Operation `description` must be present and non-empty string."><![CDATA[line 13, col 9, Warning - Operation `description` must be present and non-empty string. (operation-description)]]></warning>
        </testcase>
        <testcase time="0" name="org.spectral.operation-operationId" classname="openapi">
            <warning message="Operation should have an `operationId`."><![CDATA[line 13, col 9, Warning - Operation should have an `operationId`. (operation-operationId)]]></warning>
        </testcase>
    </testsuite>
</testsuites>

Expected behavior

• testsuites/testsuite/testcase/warning is not a valid element. must be either “error” or “failure”
• testsuites/testsuite@tests should not be equal “2” - ti should be the total number of tests run
• if all tests succeeds then a file should still be outputted (which should include the number of tests passed)

Additional context I would love to work on fixing this bug, but for now its here for reference 😉

[CasperJ]

I think we need to understand that there is a difference between formatting for console and JUnit/xUnit/mstest etc.

Humans look at console and that can show hints and other good recommendations. JUnit is usually read by machines aka. build systems which uses it to decide if it should fail or pass a build. So in this context hints doesn’t make sense as it wouldn’t know what to do with it 😃

In JUnit, if you pass the testsuites then it would “just” show “green” 123/123 tests passed. (That’s pretty much whats in the XML)

In case of a failure it would report “red” 122/123 tests passed and then the details of the error (incl. stacktrace, tracing etc. if present)

So what I’m trying to say is that what goes to the console vs. to a file doesn’t need to have the same information in it.

xUnit has the same construct in their schema: https://xunit.net/docs/format-xml-v2 => errors are documented - successes are only counted

[philsturgeon]

That makes sense. That’s why I’m fine with going either way, I suppose testing systems either work and it’s a +1 or they fail and that’s a problem which has more details.

I think we should respect the fail severity and let folks decide if error or warning is what should be shown or trigger a failure, and the format itself can just trim out anything lower as they’re not relevant to the format.

– Phil Sturgeon @philsturgeon

On Sep 30, 2019, at 21:17, CasperJ [e-mail] wrote:

I think we need to understand that there is a difference between formatting for console and JUnit/xUnit/mstest etc.

Humans look at console and that can show hints and other good recommendations. JUnit is usually read by machines aka. build systems which uses it to decide if it should fail or pass a build. So in this context hints doesn’t make sense as it would know what to do with it 😃

In JUnit, if you pass the testsuites then it would “just” show “green” 123/123 tests passed. (That’s pretty much whats in the XML)

In case of a failure it would report “red” 122/123 tests passed and then the details of the error (incl. stacktrace, tracing etc. if present)

So what I’m trying to say is that what goes to the console vs. to a file doesn’t need to have the same information in it.

xUnit has the same construct in their schema: https://xunit.net/docs/format-xml-v2 => errors are documented - successes are only counted

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.