Regression: ESLint is slow generating file lists for directories with large dot folders, e.g. Mercurial repositories, even when not including dot files

Tell us about your environment

• ESLint Version: 3.12.2 / 3.4.0
• Node Version: 7.7.2
• npm Version: 4.1.2 MacBook Pro Mid 2015.

What parser (default, Babel-ESLint, etc.) are you using? default

Please show your full configuration:

Please see https://dxr.mozilla.org/mozilla-central/source/

What did you do? Please include the actual source code causing the issue.

Run eslint on the repository, and compared the timings. For a full run on our repository, a git cinnabar checkout runs in 2m 13. For a Mercurial checkout it runs in 3m 45.

What did you expect to happen?

The repositories should take about the same time - the repository format shouldn’t matter.

This seemed to be regressed in version 3.5.0 by #6844.

What actually happened? Please include the actual, raw output from ESLint.

The Mercurial has a large .hg folder (320268 files on my checkout). Whereas a git checkout has a much smaller .git folder (3857 files).

If the repository is checked out via Mercurial, then for an eslint run, the start of the debug looks like:

$ ./tools/lint/eslint/node_modules/.bin/eslint --debug .
  eslint:cli Running on files +0ms
  eslint:glob-util Creating list of files to process. +26ms
  eslint:ignored-paths Looking for ignore file in /Users/mark/dev/mozilla-central +3ms
  eslint:ignored-paths Loaded ignore file /Users/mark/dev/mozilla-central/.eslintignore +0ms
  eslint:ignored-paths Adding /Users/mark/dev/mozilla-central/.eslintignore +0ms
  eslint:cli-engine Processing /Users/mark/dev/mozilla-central/accessible/jsat/content-script.js +52s

If the repository is checked out into a git repository (via git-cinnabar), then the eslint run looks like:

$ ./tools/lint/eslint/node_modules/.bin/eslint --debug .
  eslint:cli Running on files +0ms
  eslint:glob-util Creating list of files to process. +24ms
  eslint:ignored-paths Looking for ignore file in /Users/mark/dev/gecko +3ms
  eslint:ignored-paths Loaded ignore file /Users/mark/dev/gecko/.eslintignore +0ms
  eslint:ignored-paths Adding /Users/mark/dev/gecko/.eslintignore +0ms
  eslint:cli-engine Processing /Users/mark/dev/gecko/accessible/jsat/content-script.js +7s

The +52s -> +7s is quite a big difference. If I include all the extensions that we process via --ext then this gets even worse (2m vs 12s).

#6844 enabled parsing of dot files, and enabled searching for them via GlobSync. If, as a temporary test, I change the glob options to set dot to false, then the timings are about the same for both repositories.

https://github.com/eslint/eslint/blob/d6150e300e86e2e55a6a3d048dfb6072ac2b6489/lib/util/glob-util.js#L167

[not-an-aardvark]

Removing this from the milestone because it doesn’t need to go in a major release.

[IanVS]

I think I’ve figured out what’s going on here, and will have a PR shortly.

As @alberto correctly pointed out, we do already ignore dotfiles and dotfolders by default. But we aren’t correctly preventing globs from traversing through dotfolders. We had a PR a while back to improve our performance by not returning all of the files within node_modules from the glob (https://github.com/eslint/eslint/pull/6783), and this will be similar.

@Standard8 the reason that adding .hg to your ignore file improved performance is because we are preventing glob traversal through custom ignore patterns and node_modules, but not dotfolders. 🤦‍♂️ After my PR, I would expect you to be able to remove .hg from your ignore file without a performance hit.