Rule options from implicitly included plugins are not validated

Tell us about your environment

• ESLint Version: v5.15.3
• Node Version: v11.12.0
• npm Version: 6.9.0

What parser (default, Babel-ESLint, etc.) are you using? default (Espree)

Please show your full configuration:

Configuration

{
    "root": true,
    "env": {
        "commonjs": true,
        "es6": true,
        "node": true
    },
    "extends": ["eslint:recommended", "plugin:node/recommended"],
    "globals": {
        "Atomics": "readonly",
        "SharedArrayBuffer": "readonly"
    },
    "parserOptions": {
        "ecmaVersion": 2018
    },
    "rules": {
        "node/exports-style": ["error", "invalid-option"]
    }
}

What did you do? Please include the actual source code causing the issue, as well as the command that you used to run ESLint.

const none = require("./doesnt-exist");

$ ./node_modules/.bin/eslint .

What did you expect to happen?

I expected ESLint to report a validation error like so:

$ ./node_modules/.bin/eslint .
Error: /Users/jarrodldavis/source/repos/repro-eslint-validate-issue/explicit-plugin/.eslintrc.json:
	Configuration for rule "node/exports-style" is invalid:
	Value "invalid-option" should be equal to one of the allowed values.

    at validateRuleOptions (/Users/jarrodldavis/source/repos/repro-eslint-validate-issue/explicit-plugin/node_modules/eslint/lib/config/config-validator.js:133:19)
    at Object.keys.forEach.id (/Users/jarrodldavis/source/repos/repro-eslint-validate-issue/explicit-plugin/node_modules/eslint/lib/config/config-validator.js:176:9)
    at Array.forEach (<anonymous>)
    at validateRules (/Users/jarrodldavis/source/repos/repro-eslint-validate-issue/explicit-plugin/node_modules/eslint/lib/config/config-validator.js:175:30)
    at Object.validate (/Users/jarrodldavis/source/repos/repro-eslint-validate-issue/explicit-plugin/node_modules/eslint/lib/config/config-validator.js:262:5)
    at loadFromDisk (/Users/jarrodldavis/source/repos/repro-eslint-validate-issue/explicit-plugin/node_modules/eslint/lib/config/config-file.js:544:19)
    at Object.load (/Users/jarrodldavis/source/repos/repro-eslint-validate-issue/explicit-plugin/node_modules/eslint/lib/config/config-file.js:587:20)
    at Config.getLocalConfigHierarchy (/Users/jarrodldavis/source/repos/repro-eslint-validate-issue/explicit-plugin/node_modules/eslint/lib/config.js:240:44)
    at Config.getConfigHierarchy (/Users/jarrodldavis/source/repos/repro-eslint-validate-issue/explicit-plugin/node_modules/eslint/lib/config.js:192:43)
    at Config.getConfigVector (/Users/jarrodldavis/source/repos/repro-eslint-validate-issue/explicit-plugin/node_modules/eslint/lib/config.js:299:21)

What actually happened? Please include the actual, raw output from ESLint.

The validation error is ignored and ESLint continues as if the plugin option is valid

$ ./node_modules/.bin/eslint .

/Users/jarrodldavis/source/repos/repro-eslint-validate-issue/implicit-plugin/index.js
  1:7   error  'none' is assigned a value but never used  no-unused-vars
  1:22  error  "./doesnt-exist" is not found              node/no-missing-require

✖ 2 problems (2 errors, 0 warnings)

---

I’ve created a reproduction repository that demonstrates the difference in plugin rule validation behavior when implicitly vs explicitly including a plugin.

I believe I have narrowed it down to this code:

https://github.com/eslint/eslint/blob/a6168f85f9017332777b2bac5af8c4a979e06298/lib/config/config-file.js#L541-L552

https://github.com/eslint/eslint/blob/a6168f85f9017332777b2bac5af8c4a979e06298/lib/config/config-validator.js#L170-L173

It validates the local config (including rule options) before applying config extensions, so the plugin and its rules aren’t loaded into the linter until after the rules enabled by the local config are validated. Since the rules aren’t present at the time of validation, validation is silently skipped.

[mysticatea]

I confirmed that this reproduced.

And yes, #11546 fixed this bug because it has separated config schema validation and rule option validation. In #11546, ESLint validates config schema when each config file was read and validates rule options and environments when the configuration was determined for each source file (i.e. after all cascading config files, shareable configs, plugins, and parsers are loaded).

[fvhockney]

Great. #11546 seems to fix the issue for --fix-type as well. I’ll keep my eyes peeled for the releases.