System limit for number of file watchers reached

Describe the bug

npm start ends with a cryptic error due to exceeding the amount of files the system can watch:

Starting the development server...

events.js:170
      throw er; // Unhandled 'error' event
      ^

Error: ENOSPC: System limit for number of file watchers reached, watch '/home/u/work/some-repo/public'
    at FSWatcher.start (internal/fs/watchers.js:165:26)
    at Object.watch (fs.js:1274:11)
    at createFsWatchInstance (/home/u/work/some-repo/node_modules/chokidar/lib/nodefs-handler.js:37:15)
    at setFsWatchListener (/home/u/work/some-repo/node_modules/chokidar/lib/nodefs-handler.js:80:15)
    at FSWatcher.NodeFsHandler._watchWithNodeFs (/home/u/work/some-repo/node_modules/chokidar/lib/nodefs-handler.js:232:14)
    at FSWatcher.NodeFsHandler._handleDir (/home/u/work/some-repo/node_modules/chokidar/lib/nodefs-handler.js:422:19)
    at FSWatcher.<anonymous> (/home/u/work/some-repo/node_modules/chokidar/lib/nodefs-handler.js:470:19)
    at FSWatcher.<anonymous> (/home/u/work/some-repo/node_modules/chokidar/lib/nodefs-handler.js:475:16)
    at FSReqCallback.oncomplete (fs.js:159:5)
Emitted 'error' event at:
    at FSWatcher._handleError (/home/u/work/some-repo/node_modules/chokidar/index.js:260:10)
    at createFsWatchInstance (/home/u/work/some-repo/node_modules/chokidar/lib/nodefs-handler.js:39:5)
    at setFsWatchListener (/home/u/work/some-repo/node_modules/chokidar/lib/nodefs-handler.js:80:15)
    [... lines matching original stack trace ...]
    at FSReqCallback.oncomplete (fs.js:159:5)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! some-repo@0.1.0 start: `react-scripts start`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the some-repo@0.1.0 start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/u/.npm/_logs/2019-08-30T09_50_59_595Z-debug.log

Did you try recovering your dependencies?

Yes

Which terms did you search for in User Guide?

exclude blacklist folders directories watch watched

Environment

  System:
    OS: Linux 5.1 Manjaro Linux undefined
    CPU: (16) x64 AMD Ryzen 7 2700X Eight-Core Processor
  Binaries:
    Node: 11.15.0 - /usr/bin/node
    Yarn: Not Found
    npm: 6.10.3 - /usr/bin/npm
  Browsers:
    Chrome: Not Found
    Firefox: 68.0.1
  npmPackages:
    react: ^16.8.1 => 16.8.1 
    react-dom: ^16.8.1 => 16.8.1 
    react-scripts: 2.1.3 => 2.1.3 
  npmGlobalPackages:
    create-react-app: Not Found

Steps to reproduce

1. Create a new app with create-react-app
2. Add dependencies so that the total amount of files in the project (including node_modules) is larger than the file watch limit (cat /proc/sys/fs/inotify/max_user_watches)

My project doesn’t even have many dependencies. Mostly added with npm i apollo-boost graphql react-apollo -S. And it already created three times as many files as the maximum currently allowed by the system to watch. I would expect this problem to be quite common on some systems.

Expected behavior

I am aware of the solutions provided in #2549 and #6068.

Asking users to increase the limit only pushes the problem further without resolving it. And in my opinion is a workaround for a bad design. It also provides a less than ideal user experience and unnecessarily raises the bar for newbie developers.

The npm start script should either:

1. Not watch all files in the project by default, only project files.

The script can then ask the user to restart the app after updating dependencies and show info to how to enable watching node_modules if they want to recompile automatically after updating dependencies. From my experience it’s better to restart the application after dependencies have been updated rather than rely on watch due to problems with watching and compiling source files in linked npm modules (if used of course).

1. Watch all files but detect when the amount is beyond system limits.

The script can show a warning to the user telling them there is a problem and how to resolve it, then fall back to not watching node_modules until it’s resolved. Detecting how many files is in the project is quite quick:

# time find . | wc -l
45601

real    0m0.082s
user    0m0.021s
sys     0m0.064s

1. Watch all files but there should be another script similar to start that skips watching node_modules.

Providing such a script, say quick-start, could also serve as a quick example how to exclude certain folders from the watcher should the user opt to do that.

In any case the scripts documentation and/or troubleshooting and/or Advanced Configuration guide should mention that the start script will attempt to automatically watch all files in the project, including node_modules. Then it should be documented how to enable or disable (depending on what’s the default behaviour) watching specific folders, including how to enable or disable watching node_modules.

Actual behavior

Application doesn’t start. Only a cryptic error is shown to the user. For less advanced users it’s not possible to figure out what the problem might be without searching the internet and learning all the theory about inotify, kqueue (FreeBSD or Mac) and system limits. They may try various snippets of code available on Stack Overflow or other resources to increase the system limit, which may or may not work due to differences between systems.

[obendesmond]

I have had this error too. Try

$ echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf and $ sudo sysctl -p

This document helped me https://github.com/guard/listen/wiki/Increasing-the-amount-of-inotify-watchers#the-technical-details

[amiramix]

I think the resolution to this if any is to add a bit of documentation in our troubleshooting page linking or explaining how to increase the limit if running into this error.

How to increase the limit is easy to lookup on the internet (it’s even available above). But it’s like asking someone to buy a better phone when they complain that the web page is too big to render.

Instead of fixing symptoms, the proper fix would be to fix the root cause - not adding node_modules files to the watcher.

Best fix would be to expose the watcher configuration where one could define to ignore any folders or files of their choosing, not just node_modules.

I tried to do that but I couldn’t find where chokidar is taking its configuration from (see my earlier comments). If you could point me to the right direction I would be happy to do the fix.