Watchpack 2.0.0-beta.13 fails on large codebase with "too many open files" error

With Webpack 4 and Watchpack 1.x we avoided this error by applying a patch to Watchpack to use sane instead of chokidar, which in combination with Facebook’s watchman worked for us.

However Webpack 5 + Watchpack 2 uses Node native fs.watch which seems to end up in the same “too many open files” issue as we had before with chokidar. After a build with Webpack dev server the console just starts repeating:

Watchpack Error (watcher): Error: EMFILE: too many open files, watch

There are known existing bugs with Node’s fs.watch and many files: https://github.com/nodejs/node/issues/29460

I can work around this by using poll instead for testing - but that’s not sustainable due to the CPU overhead.

Would there be any interest in supporting watchman as a watcher “natively” to avoid another dependency on something like sane? If not, what would your recommendation be for large projects?

[sokra]

I think I fixed the issue.

I added a merging algorithm which merges watchers into recursive watchers when reaching the watcher limit.

I wrote a test and got the same results as you, only OSX has this limit of watchers. The test passes now with the new algorithm.

[sokra]

Seems like it’s using some “other” limit we are not aware of. I also found launchctl limit maxfiles, maybe you can try printing that limit.

I think a good solution is to merge watchers to recursive watchers when supported until we stay within a limit of watchers (I think 500 would be fine, so you can technically run 8 processes with watchpack). As it seems like only osx is the problem and it supports recursive watchers that should work fine.

I want to avoid recursive watchers by default as they 1. are not always supported, 2. watch too much and we don’t know how deep your directories are, which could have performance implications.