Specify max-warnings per rule

The version of ESLint you are using.

v3.13.1

The problem you want to solve.

When adding/changing rules, sometimes you can end up with lots of errors in your codebase ( especially if it is large ). max-warnings allows you to set a threshold so you and other team mates can gradually reduce it.

The problem with a global threshold is that you lose granularity of which offenses are taking place.

This is also specially problematic when maintaning a high threshold in future changes in the codebase. Ex:

• Repo has max-warnings threshold of 2000. 🙀
• Somebody makes a PR, and CI fails because ESlint says there are now 2002 warnings, instead of 2000.
• The developer that made the PR now has to dig through a very long ESlint output, trying to figure out which new offenses has been introduced.

Another edge-case of losing granularity is that sometimes fixes and new offenses cancel each other, resulting in a very similar threshold. Ex: I can fix 20 errors regarding semicolons, while at the same leaving 20 console.log statements. CI does not fail because the number of warnings is the same :trollface:

Your take on the correct solution to problem.

Ability to specify a max-warnings threshold per rule.

Before ESlint, one thing that has worked in my company ( cc @fcsonline ) is to have a list of threshold per offense. We spotted some bad practices, so with some grep magic help, we were able to detect them an specify a maximum threshold per bad practice.

This came very handy and maintainable in the everyday job, because developers could easily spot what bad patterns they were introducing. Thresholds gradually decreased, while avoiding too much yak shaving. We also made sure that CI failed if the thresholds needed to be updated, showing a motivational message ( imagine BETTER WORLD! no-console warnings found 23 ( current threshold 25 ). Please update threshold =] ).

This also gives exposure in code review, of which bad practices you are introducing, and which ones you are fixing.

.eslintrc or .eslintmaxwarnings

{
...

"max-warnings" {
  "no-console": 30,
  "no-extra-boolean-cast": 12,
}
...
}

Non specified max-warnings are assumed to have a max-warnings of 0.

Exit status is 1 if some of the specified max-warning threshold is surpassed.

Thoughts?

[platinumazure]

The idea is interesting, but I’m not 100% sure this should belong in core. In particular, the note you had about reporting when a rule had dropped under a threshold seems like a project-specific CI concern.

So I personally think your best bet is to implement a Node wrapper around ESLint using our CLIEngine API, and that wrapper could read and maybe even write its own configuration. You could use the executeOnFiles method to get the errors, optionally call outputFixes to apply the suggested auto-fixes, and then use the results to decide if a rule has too many errors, or if it has few enough that a warning should be emitted for reducing the threshold.

That’s my suggestion, but maybe others on the team would be interested in integrating this into core. Let’s see what they think.

[aboyton]

https://github.com/IanVS/eslint-nibble might also be of interest to you.