Make CLI not to return non-zero exist code when only warnings occur

User story.

As a continuous integration user, I can provide a parameter to CLI, so that the CI pipeline won’t fail when there is no error but only warnings.

As a continuous integration user, I can provide a parameter to the CLI, so that only results of a certain severity will be returned as output.

Describe the solution you’d like

Updated 2019-07-16 by @philsturgeon

Two new CLI options:

1. --fail-severity=warn - enum[hint, info, warn, error] - change the level of results which come out of Spectral.
2. --display-only-fail-severity-results - boolean - spectral will error if results of any level at all are reported. We should make that configurable.

We should probably not change the default behavior in this as its aiming for v4.1. Place a TODO note in the code to change the default to warn in v5.0.

[soulcutter]

--fail-level is my favorite so far. It might be nice to have some consistency between the fail flag and the output flag, sooo… --output-level maybe?

[aleung]

Though you all prefer boolean flag, let me share my use case of ESLint --max-warnings. No going to convince somebody but explain why ESLint provides this kind of option.

Linter wasn’t used in my team from day 0. When I introduced it we already had tens of projects and their quality was no so good. You can imagine that the linter reported a large number of errors and warnings. I forced the team to fix errors immediately but the project schedule won’t allow us to spend too much time to fix all warnings – I had carefully define the ruleset to limit the number of error rules. But if there is no any restriction on the number of warnings, the team won’t take care of them. So I set max warnings in the CI pipeline to make sure at least the number of warnings won’t go up. And also I decreased the number gradually, so that developer had to fix some warnings in each commit but not too many to impact his/her progress. After 2~3 months, the code quality finally reaches a satisfactory status.

Even though, I never set the max warnings to zero but give it a reasonable value – of cause you can say it’s arbitrary. We adjust the value by retrospective of overall product quality. We do not force to fix all warning because in some cases there is reason to break the rule otherwise there will be other negative impacts. (BTW, in ESLint/TSLint there is option to temporarily disable a rule for a line, but it’s impossible to do it in Spectral). Working in software enterprise, we have to compromise and get balance between quality, workload, and schedule.