windows chunking git lock race condition when run through husky

Description

When the chunked files are split up into enough groups the chance of a race condition on calling git add increases. If chunk a calls git add while chunk b is adding there will be a git.lock file which will make the process fail.

tested on latest version of husky and lint-staged but still occurs. Originally experienced with lint-staged: 10.0.9 husky: 3.0.9

Tested with lint-staged: 10.1.2 husky: 4.2.3

Issue appears to be in gitWorkflow.js

await Promise.all(
      // stagedFileChunks includes staged files that lint-staged originally detected.
      // Add only these files so any 3rd-party edits to other files won't be included in the commit.
      this.stagedFileChunks.map(stagedFiles => this.execGit(['add', '--', ...stagedFiles]))
    )

There is no way to make sure that the git add commands are concurrent for chunks to prevent having a git lock scenario.

Steps to reproduce

Merged 1k+ files where there are lint changes to be applied call through husky on windows

configuration: “husky”: { “hooks”: { “pre-commit”: “lint-staged” } }, “lint-staged”: { “.{ts,tsx}": “eslint --quiet --fix --cache”, ".scss”: “stylelint --syntax=scss” },

Debug Logs

expand to view

Applying modifications... [failed]
→ fatal: Unable to create '{repo}/.git/index.lock': File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
Restoring unstaged changes to partially staged files... [started]
Restoring unstaged changes to partially staged files... [completed]
Cleaning up... [started]
Cleaning up... [completed]

Environment

OS: Windows 10 Node.js v10.16.1 lint-staged: latest 10.x

Previously there was a bug pre-10 where windows would fail due to command-line args length Upgraded to 10 to fix this but keep hitting async issues with git when merging multiple chunks. The issue at the time of raising was a merge of 47 chunks.

[iiroj]

I actually managed to implement this feature in the same PR!

Since we already go through all (chunked) tasks in runAll, we can add all matched files to a Set, which will by definition contain unique items.

Then I just pass this set instead of the full chunkedFilepaths to GitWorkflow, chunk it, and then run git add “serially” for all the chunks.

[iiroj]

Thanks for the suggestion. This particular place is the only one where the entire stagedFileChunks is used after generating the tasks, which need all files for matching.

It would certainly be possible to create a (smaller) list of matched files, and then chunk that for the git add part. I haven’t done it because it has not been a priority, and to be honest I didn’t think about it.

I appreciate your help!