Proposal: add option to include comments in token store methods

What version are you using? Master

The problem I’m trying to solve On instantiation, the sourceCode object creates a token store and then creates public methods that call through to the token store methods. Since this does not include comments, we have had to create a separate store that includes comments and then create additional methods, as seen here.

Working on https://github.com/eslint/eslint/pull/7696, I ended up needing to expose another method from the tokensAndCommentsStore, which doesn’t feel good, is confusing, and makes it hard to maintain.

Proposed solution I would like to propose unifying this API by adding the ability to include comments in the results with { includeComments: true } as the last parameter for any applicable token store method.

So, instead of having

sourceCode.getTokenBefore(node, skip);
sourceCode.getTokenOrCommentBefore(node, skip);

we would have

sourceCode.getTokenBefore(node, skip);
sourceCode.getTokenBefore(node, skip, { includeComments: true });

Two possible implementations immediately jump out to me:

• Having sourceCode create two token stores as it does now and create public proxy methods that call the method on the correct store using the API I’ve proposed above.
• Updating createTokenStore to take in ast.comments as a second parameter, having it create two separate token lists internally, and explicitly update all its public methods with the additional { includeComments: true } parameter. This is my preference, as I think it’s clearer and makes more sense conceptually for the token store to manage this internally.

This can be done in a non-breaking way, as we can make getTokenOrCommentBefore and getTokenOrCommentAfter call through to the correct methods, and in any cases where methods are invoked without the { includeComments: true } argument, nothing will change.

I’m very much willing to work on implementing this 😃 Would love it if some JSCS folks could chime in, as I believe this is how JSCS handled this.

[kaicataldo]

@nzakas Having made the issue, I obviously disagree. Making all the currently available token iterators have the ability to include comments gives rule creators more tools. Right now, they would have to request that we add a new method to core every time they need new functionality, as I had to while working on the PR that led me to this.

What if we made this change in a non-breaking change (implementation details in the proposal)? The current two “tokenOrComment” methods aren’t documented anywhere yet, so I think now is the time to remove these if that’s something we want to do (we can deprecate them and still have them call the correct underlying token store method and then remove them in the next major version).

I think the alternative to this is to combine the two token stores and then to duplicate all the relevant methods in token store up front - one for tokens and one for tokensAndComments - rather than adding them in piecemeal as needed.

Though I’m advocating for the former, as I think it’s a nicer API (and avoids having to duplicate a bunch of methods!), I’m not against the latter if it’s easier to gain consensus on that. Either would be an improvement of the code and give us and rule authors clarity on how to iterate over comments 😃

Thoughts?

[kaicataldo]

@eslint/eslint-team Any other thoughts?

@mysticatea Sounds like your task of refactoring doesn’t need to be blocked by this (and I don’t think needs to be tracked in this issue, if it truly is a refactor). We can always add more methods afterwards, if that’s what the decision here is.