Idea: release a wrapper script that helps you run decaffeinate

decaffeinate is great, but it really feels like a building block. To help manage the process of running decaffeinate, I ended up building a Python script that has accrued quite a few features over time. It seems like it would be useful if decaffeinate officially had something similar: a wrapper tool that makes it easier to run and manage a transition of a large codebase. I could also hear the argument that it should be part of the core decaffeinate CLI rather than a separate script, although I like the split where decaffeinate is a focused coffee-to-JS tool and the wrapper script is allowed to make more assumptions and bring in other tools.

Here’s the script copied from the main internal repository at Benchling: https://gist.github.com/alangpierce/48decab49ff76c5bed55203f1cbdf0f7

Here’s the full set of steps that it does:

1. It prompts you to create a file called files_to_decaffeinate.txt, where each line is a path to a .coffee file.
2. It does some sanity checks: the files you specify need to exist, the equivalent .js files can’t already be there, etc.
3. If the file exists, by default, the script just does a “dry run” and checks decaffeinate on each of these files and generates a report at the end. Currently it does this by running decaffeinate on the .coffee file and rming the resulting .js file when it’s done.
   • For each file processed, it prints out a line with either OK or ERROR followed by an error code. It does string matching on the decaffeinate stdout to match some common errors I’ve run into, which helps me track progress and figure out priority for bug fixes.
   • For any failures, it prints out a command that can be used to open the file contents in the repl. The suggested workflow is to tweak the .coffee files in the repl to be decaffeinate-compatible, copy the changes back and commit them, then rerun everything again until decaffeinate works on all specified files.
   • If there were no errors, it says that it’s ok to proceed with the “convert” step.
4. When running the convert step, first it does a full dry-run to make sure the conversion will work before it makes any actual changes.
5. It backs up all .coffee files as .original.coffee files, which I have under .gitignore. This is useful later when I want to compare the old and new code side-by-side.
6. It generates a git commit that just does “git mv” from every .coffee to .js file, which preserves git history as described in the conversion guide. In this commit and all other generated commits, the author name is changed to “Decaffeinate” (but the email is the same) so that people using git blame can be aware that the file was created by the decaffeinate process and not authored by me.
7. It moves the .js files back to .coffee and runs decaffeinate, then removes the .coffee files, and commits that as another git commit.
8. It runs the arrow-function jscodeshift script to convert function expressions to arrow functions as much as possible, since that’s consistent with our style.
9. It runs a custom jscodeshift script that I wrote to recognize our convention for writing React elements and convert them to the more typical React.createElement format.
10. It runs the create-element-to-jsx jscodeshift script to convert React.createElement into JSX.
11. It puts /* eslint-env mocha */ at the top of every test file, since that’s how we make eslint aware that it’s a mocha file.
12. It runs eslint --fix on all files. This ends up doing a number of automatic fixes for our particular style rules: using arrow functions when possible, using const when possible, using trailing commas, fixing various little spacing issues, and probably others.
13. For any remaining lint failures, it puts an eslint-disable comment at the top of the file that specifically disables the rules for all of those failures. That makes it possible to submit the file for code review and also prevents most new lint errors from being introduced.
14. It adds a TODO comment to the top of these files that says to clean up any remaining lint failures and look through the file for any style or correctness issues.
15. It prints out instructions for what to do next, including the details on how to submit the code review to Phabricator and how to land the commit. The intention is that there will immediately be a follow-up commit to fix the remaining style issues.

Some of these are Benchling-specific, but it seems like most of the steps would be useful to anyone. git and eslint are probably common enough that just about anyone is using them these days.

I’m interested to hear how other people run decaffeinate. Do you typically run decaffeinate directly from the CLI? Do you just use the repl? Do you go through the effort of keeping source control history across the transition? Have you tried doing the automatic post-cleanups like I’m doing? Is there anything you’ve found useful that’s missing from my script?

In my case, I’ve been using this script with about 5-10 files at a time, and I’ve done the full process on about 20-30 files so far. but I’m hoping to get everything in a stable enough point that I can confidently convert them in larger batches. The codebase I’m working in has about 1300 CoffeeScript files, so I’d rather not have to split the conversion process up into hundreds of commits.

I guess one concern with a script like this is it may be too specific; it might be the type of thing that’s better to write yourself for your exact requirements. Still, I bet some of these steps can be released in a way that can be used broadly, and I bet the project-specific things could be specified as a configuration (e.g. a list of jscodeshift scripts, or scripts in general, to run). Also, I’ve recently been running into some tricky decaffeinate issues that may require cross-file knowledge (bug reports coming soon), so this tool could handle that type of thing while keeping the core decaffeinate as a tool that just needs to operate on one file at a time.

I’ll probably start hacking on a flexible JavaScript port of my script pretty soon regardless (I’ll probably call it decaffeinate-orchestrator, but I’d be open to other names), but would be good to hear what others think and if there are any other ideas for this type of thing.

[alangpierce]

Ok, I just published an initial version of this kind of tool under the name bulk-decaffeinate. (I gave up on the name decaffeinate-orchestrator after trying to type it a few times.) This is my first time publishing something to npm, so hopefully I did it right.

Currently, it’s useful for evaluating how ready a codebase is for decaffeinate (it doesn’t yet help with actually converting files). It runs decaffeinate using 4 parallel workers on all .coffee files in a directory, records any errors that comes up, and creates a log file with all failed files and their error messages. It then makes it easy to open all of those failed files in the repl. See the README for an example of using the script to try decaffeinate on the Hubot source code.

Speaking of which, it seems like it would be cool if we measured progress by seeing how well decaffeinate does in the most popular CoffeeScript projects on GitHub: https://github.com/trending/coffeescript . I looked through some of the failures and it looks like there’s some low-hanging fruit. It seems like it would especially help decaffeinate’s credibility if we got it working well on major open source projects.

The code for bulk-decaffeinate is in a GitHub project under my user: https://github.com/alangpierce/bulk-decaffeinate . I tried to make the style and tooling similar to decaffeinate. If it gets more mature/useful, we could maybe move it to the decaffeinate org and officially recommend it.

[eventualbuddha]

Sounds good!