Fixing autofix

There have been a few issues recently with people asking to add more autofixing. This is definitely something we want to address, but we can’t do it with the way autofixing works right now. This (unfortunately long) issue is meant to outline what we currently have, its limitations, and start towards thinking about how to fix autofix so we can use autofix in as many places as possible.

Intro

When ESLint first implemented autofixing, our belief was that it was primarily useful for making small changes, mostly whitespace and anything that could be contained within a single token. Most of the complaints people had about ESLint rules was that they hated going through and fixing alignment or adding/removing semicolons but loved the error catches. So naturally, I thought if we could get the whitespace and other small autofixes in, that would make people happy. It did, it’s just that then we started getting requests to autofix more rules and we realized that the current system doesn’t scale well.

How it Works Today

Today’s autofix is based on text and is latched onto how ESLint has always worked:

1. ESLint makes a single traversal through the AST of the file.
2. Rules listen for different nodes and evaluate the AST as it goes, producing messages when it finds something wrong.
3. Rules can optionally specify a way to fix each message it reports. The fix is indicated by range offsets (the same as used in the range object on AST nodes). This fix is not guaranteed to be applied and will not even be attempted until after the complete traversal is finished.
4. After the traversal is complete, all of the proposed fixes are sorted in descending order of range location. So a fix at location 10 comes before a fix at 5.
5. The fixer attempts to apply the fixes in this order. If two fixes overlap in range, then only the first is applied. This continues until all potential fixes have been processed.
6. If a fix is applied, then that message is removed from the lint results of the file.
7. The remaining lint results are presented to the user.

Because of the single traversal, we can’t apply the fixes as we go because it would affect other rules as it goes. For instance, if we changed a let to a const, that then pushes the ranges of everything in the tree by two, so when does that calculation happen? Also, each rule is triggered multiple times at different nodes, and many of the rules try to track things in between those triggers - the built-in assumption is that the tree looks the same as the first time the rule is triggered.

Problems

Due the single traversal mode, we have several problems:

1. It’s possible that not all fixes will be applied.
2. It’s possible for a fix to be applied that violates a different rule. For instance, one rule could insert a comma while another requires a space after a comma. The rule requiring the space has no idea that a new comma was added because it’s done after the rule has already executed.
3. It’s possible for a fix to change scope evaluation. For instance, changing var to const has implications based on the functions or block statements using it. That can invalidate a whole host of rules because scope calculations are done once, before the traversal.
4. It’s possible for a fix to introduce an error into the program by changing the meaning of the code.

As a result of these problems, we decided to limit what we’d allow as fixes in core rules to whitespace and other small changes. Semicolons were a big pain point, so even though they aren’t whitespace, we determined the value of fixing these was high enough to be worth any potential other problems. The same with string literals. We actually removed autofixing for === after we realized it would change the meaning of the condition and cause a runtime error. We then decided that any new autofixes would just be whitespace to avoid any other complications.

Where I Went Wrong

Looking back, I think I made several design decisions that were incorrect. These are the things I think I got wrong (and I use “I” here since I came up with the design):

1. Autofix mode should just do fixes and not worry about unfixable rules. Right now it acts just like linting, outputting any messages it didn’t fix. That seems unnecessary.
2. The text-based fixes were a naive approach to autofixing that was targeted at whitespace fixes as a primary use case. It’s clear text-based fixes aren’t scaleable.
3. Trying to fit autofixing into the same single traversal system as the linting is too limiting.

The Future

So now that I’ve explained the current state of stuff, I think we have an opportunity to fix it (no pun intended). This will likely be a big breaking change, but hopefully will be better for what people really want to use autofix for. I don’t know the implementation details, but I have a high-level of idea of where I’d like to see autofix go.

Requirements

1. Autofix mode just does fixes and does not do any unfixable rule linting at all. (That means --fix doesn’t output any linting problems.)
2. Autofix logic for each rule is contained with the rule file that does linting, not it another file.
3. Autofix logic in rules is declarative and doesn’t force rules to do something like if (mode === "fix") { fix() }. Basically, we can’t throw away the way rules work today, there are too many core rules and plugins with rules to force everyone to rewrite in a dramatic way.
4. Multiple autofix passes can be done on the same file to ensure as many fixes as possible are applied.
5. Regardless of fixes, ESLint must never output a file with illegal syntax as the final result.
6. Autofix should work for custom nodes that aren’t part of the ESTree spec (I’m thinking about Flow, TypeScript, and other derivatives that add custom nodes into an ESTree AST).

Ideas: Config

One of my ideas is that you should be able to specify which rules you want autofix to apply for, so in your config file, you’d do something like:

rules:
    semi: "fix"
    quotes: ["fix", "double"]

This would be an extension of the work done in https://github.com/eslint/eslint/issues/3626 to allow “off”, “warn”, and “error” as options instead of 0, 1, and 2.

I think this would be a big win because it gives us a good idea about how to save time: during fixing, we just load the rules marked as “fix” and ignore the rest. That should cut down on the runtime (so long as not every rule is fixable!)

Open Question: When not running in fix mode, what does “fix” mean? Is it “warn” or “error”?

Concerns

Some other concerns I have with changes are:

1. Is it possible to do multiple fixes in a single traversal? Or do we need to do one rule’s fixes at a time?
2. Do we need to do nonwhitespace rules first and whitespace rules second? Or do we just bite the bullet and say we always need at least two passes per file to get as many fixes as possible? (Or just keep doing passes until all fixes are applied?)
3. How will this work with code path analysis? Changing nodes around affects code paths dramatically.
4. How will this work with escope and scope evaluation? Will we have to run scope analysis multiple times? How will that affect performance

Next Steps

This is just the beginning of the conversation, there are a ton of details that need to be hashed out. I just wanted to share this to ensure everyone that we are thinking about it and we do want to make improvements.

[nzakas]

Our informal Twitter poll (https://twitter.com/geteslint/status/713112461998628865) shows that people are overwhelmingly in favor of applying more fixes even if that means a slower experience. Given that, I’m going to experiment a bit with doing multiple passes of files. Doing multiple passes would, in the short term, allow us to fix more problems without needing to completely change the way we do fixes. I think this is a good small step to get better compatibility with JSCS 2.x.

[mikesherov]

Perf on autofix shouldn’t be a huge concern IMO because it’s not the majority use case. Unless it’s extremely slow (slow enough to be noticeable on one file), then the main use case is run once on entire codebase, then run a bunch of times (like on save) for individual files. Simplicity wins for me here personally.

With that said, whitespace fixes are a solved problem: as long as there is an interface for catching errors, that same interface can be used to fix the error. Add space, remove space, add new line, indent are all as trivial to detect as they are to fix. This solves the “no need for new code” desire that @nzakas has. Also, just keep doing passes until the number of remaining errors doesn’t go down. There’s a small chance the number can go up and ultimately end up going down, but the simplest, good enough solve is a dumb multipass as I described.

Non whitespace fixes is not an area I explored, so I’ll defer to the rest there.

In this regard though, I think the intuition of separating the whitespace fix stage from the non whitespace fix stage is a good one.