Automation plan with failOnWarning: true still ends with exit code 0 if scan produces warnings

Describe the bug I expected the automation framework parameter failOnWarning with value true to cause the execution of a plan containing it to fail, if the scan results in any warnings. That does not seem to be the case. Upon running the plan below against a target that produces warnings, the exit code remains 0. Please excuse me, if I misunderstood the meaning of failOnWarning in the first place!

To Reproduce Run the following plan against a target that produces warnings:

env:
  contexts:
  - excludePaths: []
    name: baseline
    urls:
    - http://ci-demo-master
  parameters:
    failOnError: true
    failOnWarning: true
    progressToStdout: false
jobs:
- install:
  - pscanrulesBeta
  type: addOns
  uninstall: []
- parameters:
    enableTags: false
    maxAlertsPerRule: 10
  type: passiveScan-config
- parameters:
    maxDuration: 1
    url: http://ci-demo-master
  type: spider
- parameters:
    maxDuration: 0
  type: passiveScan-wait
- parameters:
    format: Long
    summaryFile: /tmp/zap_out.json
  rules: []
  type: outputSummary
- type: report
  parameters:
    reportDir: /tmp/testing/artifacts
    template: traditional-html
- type: report
  parameters:
    reportDir: /tmp/testing/artifacts
    template: traditional-json

Expected behavior The execution of the automation plan ends with an exit code != 0

Screenshots

Software versions

• ZAP: 2.11.0
• Add-on: None except the defaults and the ones required by the plan above
• OS: Official zap2docker-stable container
• Java: 11.0.11 according to ZAP output (Found Java version 11.0.11)
• Browser: -

Errors from the zap.log file No errors, only INFO output in zap.log

Additional context

Would you like to help fix this issue? Gladly 👍

[psiinon]

Yeah, I get this too 😦 Will investigate further…

[va1entin]

Thanks a lot! Since there’s an easy workaround with the summaryFile this is probably not the highest priority so no worries 😃

A little follow-up: The zap2docker image doesn’t contain jq so here is a one-liner to achieve the same as outlined above without jq.

warn=$(grep warn zap_out.json | grep -Eo '[0-9]*')