messageId messages with placeholders fail during tests.

Tell us about your environment

• ESLint Version: 4.15+
• Node Version: 8.9.3
• npm Version: 5.5.1

What parser (default, Babel-ESLint, etc.) are you using? default

Please show your full configuration: N/A

What did you do? Please include the actual source code causing the issue, as well as the command that you used to run ESLint. Write a test where the message is a placeholder and the message is referred to via the new meta.messageId. Run the test runner against it with a messageId to match against.

lib/rules/no-foo.js

module.exports = {
    meta: {
        messages: {
            avoidName: "Avoid using variables named '{{ name }}'"
        }
    },
    create(context) {
        return {
            Identifier(node) {
                if (node.name === 'foo') {
                    context.report({
                        node,
                        messageId: 'avoidName',
                        data: {
                            name: 'foo'
                        }
                    });
                }
            }
        };
    }
};

test/rules/no-foo.js

'use strict';
var rule = require('../../../lib/rules/no-foo');
var RuleTester = require('eslint').RuleTester;

var ruleTester = new RuleTester();
ruleTester.run('no-foo', rule, {
    valid: ['bar', 'baz'],
    invalid: [
        {
            code: 'foo',
            errors: [
                {
                    messageId: 'avoidName'
                }
            ]
        }
    ]
});

npm run test

What did you expect to happen? Tests pass.

What actually happened? Please include the actual, raw output from ESLint.

  1 failing

  1) no-foo invalid foo:

      AssertionError [ERR_ASSERTION]: 'Avoid using variables named \'foo\'' === 'Avoid using variables named \'{{ name }}\''
      + expected - actual

      -Avoid using variables named 'foo'
      +Avoid using variables named '{{ name }}'

      at assertMessageMatches (node_modules\eslint\lib\testers\rule-tester.js:442:24)
      at testInvalidTemplate (node_modules\eslint\lib\testers\rule-tester.js:517:29)
      at Context.RuleTester.it (node_modules\eslint\lib\testers\rule-tester.js:581:25)

So it looks like ruleTester is comparing the value of messageId’s pre-formatted template to the post-formatted runtime error.

It seems like the ideal behavior is:

1. If your test needs to get the post-formatted runtime report message, use message in the ruleTester.
2. If your test wants to know that a specific rule ID was used to generate the error message, use mesageId in the ruleTester.

My guess is that most use cases fall under the second category, but that the first category is worth supporting as well to give access to the direct output during testing if needed.

[betaorbust]

So, everybody good if I put a PR in that satisfies https://github.com/eslint/eslint/issues/9890#issuecomment-361110406?

[not-an-aardvark]

This seems like it complicates the expected use of the RuleTester API as the rest of the attributes are independent and direct-comparison

That’s a fair point. I think data is a bit different from other properties in that the “important” keys are dependent on the message/messageId anyway. For example, some rules provide an AST node as the data property and rely on ESLint to grab the appropriate properties from it:

https://github.com/eslint/eslint/blob/084351bedb1001fc460e9acea4952b0b6139ec23/lib/rules/prefer-const.js#L300-L305

With code like that, it probably wouldn’t make sense to match against every property of node, since there are a lot of unrelated properties and the data is functionally equivalent to { name: node.name }. So either way, I think there is a coupling between message/messageId and data that doesn’t exist between other properties, so I’m fine with having the corresponding RuleTester options coupled.