[RuleTester] Test case validation

The version of ESLint you are using. Node version: v15.3.0 npm version: v6.14.2 Local ESLint version: v7.15.0 (Currently used) Global ESLint version: Not found

The problem you want to solve. The rule tester throws a type error and does not run any test case if the test case property is not a string:

import { RuleTester } from 'eslint';
const ruleTester = new RuleTester();
ruleTester.run("rule-id", rule, {
	valid: [{
		code: [
			'...',
		] // <-- missing .join('\n')
	}],
	invalid: [],
});

which results in an ugly error:

text.replace is not a function 
  at ​​​sanitize​​​ ​./node_modules/eslint/lib/rule-tester/rule-tester.js:207

Relevant code is here. I categorized this as a change as there is only little test case validation but this issue may also be seen as a bug report.

Your take on the correct solution to problem. The rule tester only fails the impacted test case with an instruction on how to correct the error. This may be expanded to other properties than the code property. Currently the rule tester only checks the structure of the scenarios (eg. valid and invalid are present) and for test cases that options are an array and that parser is an absolute path.

Are you willing to submit a pull request to implement this change? Yes

[mdjermanovic]

Makes sense to me, it would be useful to provide more helpful info. I’ll champion this.

Currently the rule tester only checks the structure of the scenarios (eg. valid and invalid are present) and for test cases that options are an array and that parser is an absolute path.

Some properties are passed through as ESLint config properties and validated as such. For example, try to specify globals: [] or env: [].

These are all RuleTester-specific properties:

https://github.com/eslint/eslint/blob/06b58096975935ec016d96dd5f333f059c270f26/lib/rule-tester/rule-tester.js#L118-L124

options are validated in runRuleForItem, errors is validated in testInvalidTemplate.

Aside from code, we can consider explicit validations for the remaining two properties, but I think we should limit the scope to only those that certainly cause throwing or failing the test if the type is invalid. Other changes would probably need a RFC.

[mdjermanovic]

Marked as accepted since this is more of a bug than an enhancement.