Handling warnings/errors for configuration and other non-rule issues

The version of ESLint you are using.

Latest

The problem you want to solve.

We are currently combining rule warnings/errors, parse errors, and some types of configuration errors in the lint messages object, separating as errors vs warnings but otherwise keeping them all together.

Since the warnings array in the lint result is counted and compared against the --max-warnings CLI option, this means that we sometimes find ourselves unable to add new configuration warnings (and other UX improvements to help our users configure and use ESLint correctly) due to not wanting to break users’ builds on semver-minor releases.

I would like to allow us more flexibility on adding warnings for configuration or UX issues as semver-minor changes. Long-term, we need to be able to do this without having any impact on the exit code for the ESLint process.

This is currently blocking eslint/rfcs#34.

Your take on the correct solution to problem.

Two steps, one short-term and one long-term. (Note: The long-term plan is somewhat out of scope for this issue specifically, because it should actually be handled by an RFC. But I am hoping that describing the long-term approach at a high level will encourage the TSC to adopt the short-term approach for now.)

• Short-term approach: No technical change, just a process change which TSC could adopt via motion in TSC meeting.

  Basically, we would allow the TSC, on a strictly case-by-case basis, to allow specific proposals (which add new warnings) to be releasable in semver-minor releases. We would adopt specific guidelines and add them to the Maintainer Guide.

  As an initial proposal for those guidelines, I would suggest something like: (1) This shall not modify our current policy for warnings generated by ESLint rules; and (2) Anyone can add an issue to the TSC agenda and ask that the TSC consider accepting the proposal as semver-minor, despite adding more warnings from a non-rule source. But, we can discuss in more detail in the TSC meeting discussing this proposal. 😄

• Long-term approach: Augment the lint result object to contain a top-level property for general lint run issues (including, but not limited to, configuration issues). This would contain warnings and errors arrays. Any errors in this object would result in exit code being nonzero (same as file-level errors, including rule errors). However, any warnings in this object would not result in exit code being nonzero. --max-warnings would not look at the warnings in this object-- it would only look at file-level warnings (ideally just rule warnings).

  Of course, this needs to be fleshed out in an RFC.

Are you willing to submit a pull request to implement this change?

Yes-- if the TSC agrees to the short-term approach, I would be willing to submit a PR to update the Migration Guide appropriately.

I could also work on the long-term RFC, but I am willing to let someone else take over that part if they are interested.

[platinumazure]

TSC Summary:

This is a follow-up from a TSC discussion on 2019/12/05 about whether configuration-related warnings could ever be added in a semver-minor release.

I would like to allow us more flexibility on adding warnings for configuration or UX issues as semver-minor changes. Long-term, we need to be able to do this without having any impact on the exit code for the ESLint process.

More background can be found in the initial issue post. Also, this is currently blocking eslint/rfcs#34.

Long-term, we should probably fix this by creating a new location in the lint result object for lint run-level issues (e.g., configuration issues). Short-term, I would like to ask the TSC to consider allowing exceptions to the semver rules, on a case-by-case basis, for specific proposals in which the value of adding warnings is high enough to potentially justify breaking the users’ build on semver-minor releases (especially if ESLint is already currently not working for the use case in question and we want to improve the UX and help users fix their issue). The process would be to nominate the issue on the TSC Agenda and ask the TSC to decide if the proposal should be accepted while adding warnings on a semver-minor release.

This process change would be captured in the Maintainer Guide.

TSC Question:

Should we adopt a policy to allow, on a case-by-case basis, TSC discussion on issues that add warnings and whether they could be allowed in a semver-minor release?

[kaicataldo]

Reopening because we still need to discuss this. @eslint/eslint-tsc Please give this a look when you get a chance!