[Bug] webpack-dev-server can't recompile: Can't open file: No file descriptors available

• I’d be willing to implement a fix

Describe the bug

We are using webpack-dev-server middleware to develop locally. The initial compilation is fine as well as the production builds (via webpack-cli -p) but as soon as I change a file the dev-server’s recompilation fails with:

ERROR in ./app/learning-center/TrainingsPage/TrackTable.js
Module build failed (from /Users/nkalinov/dev/simonapp/.yarn/$$virtual/babel-loader-virtual-04f6d5f491/0/cache/babel-loader-npm-8.0.6-ccc68d8d38-1.zip/node_modules/babel-loader/lib/index.js):
Error: Can't open file: No file descriptors available
Require stack:
- /Users/nkalinov/dev/simonapp/.yarn/cache/loader-runner-npm-2.4.0-c414104c2f-1.zip/node_modules/loader-runner/lib/loadLoader.js
- /Users/nkalinov/dev/simonapp/.yarn/cache/loader-runner-npm-2.4.0-c414104c2f-1.zip/node_modules/loader-runner/lib/LoaderRunner.js
- /Users/nkalinov/dev/simonapp/.yarn/cache/webpack-npm-4.41.5-ec6f53e9d3-1.zip/node_modules/webpack/lib/NormalModule.js
- /Users/nkalinov/dev/simonapp/.yarn/cache/webpack-npm-4.41.5-ec6f53e9d3-1.zip/node_modules/webpack/lib/NormalModuleFactory.js
- /Users/nkalinov/dev/simonapp/.yarn/cache/webpack-npm-4.41.5-ec6f53e9d3-1.zip/node_modules/webpack/lib/Compiler.js
- /Users/nkalinov/dev/simonapp/.yarn/cache/webpack-npm-4.41.5-ec6f53e9d3-1.zip/node_modules/webpack/lib/webpack.js
- /Users/nkalinov/dev/simonapp/.yarn/$$virtual/webpack-dev-server-virtual-d46dc61805/0/cache/webpack-dev-server-npm-3.10.1-34b49b6a2b-1.zip/node_modules/webpack-dev-server/bin/webpack-dev-server.js
    at new ZipFS (/Users/nkalinov/dev/simonapp/.pnp.js:27790:15)
    at ZipOpenFS.getZipSync (/Users/nkalinov/dev/simonapp/.pnp.js:32288:52)
    at ZipOpenFS.makeCallSync (/Users/nkalinov/dev/simonapp/.pnp.js:32187:17)
    at ZipOpenFS.existsSync (/Users/nkalinov/dev/simonapp/.pnp.js:31685:17)
    at VirtualFS.existsSync (/Users/nkalinov/dev/simonapp/.pnp.js:26209:24)
    at /Users/nkalinov/dev/simonapp/.pnp.js:39078:28
    at Array.find (<anonymous>)
    at applyNodeExtensionResolution (/Users/nkalinov/dev/simonapp/.pnp.js:39076:10)
    at resolveUnqualified (/Users/nkalinov/dev/simonapp/.pnp.js:39430:27)
    at resolveRequest (/Users/nkalinov/dev/simonapp/.pnp.js:39462:14)
    at Object.resolveRequest (/Users/nkalinov/dev/simonapp/.pnp.js:39524:26)
    at Function.module_1.Module._resolveFilename (/Users/nkalinov/dev/simonapp/.pnp.js:38753:34)
    at Function.module_1.Module._load (/Users/nkalinov/dev/simonapp/.pnp.js:38638:40)
    at Module.require (internal/modules/cjs/loader.js:848:19)
    at require (internal/modules/cjs/helpers.js:74:18)
    at loadLoader (/Users/nkalinov/dev/simonapp/.yarn/cache/loader-runner-npm-2.4.0-c414104c2f-1.zip/node_modules/loader-runner/lib/loadLoader.js:18:17)
 @ ./app/learning-center/TrainingsPage/index.js 47:41-64
 @ ./app/learning-center/index.js
 @ ./app/app/App.js
 @ ./app/main.js
 @ ./app.js
 @ multi /Users/nkalinov/dev/simonapp/.yarn/$$virtual/webpack-dev-server-virtual-d46dc61805/0/cache/webpack-dev-server-npm-3.10.1-34b49b6a2b-1.zip/node_modules/webpack-dev-server/client?https://localhost:8081 (webpack)/hot/dev-server.js ./app.js

ERROR in chunk main [entry]
Can't open file: No file descriptors available
Require stack:
- /Users/nkalinov/dev/simonapp/.yarn/cache/webpack-npm-4.41.5-ec6f53e9d3-1.zip/node_modules/webpack/lib/web/JsonpMainTemplatePlugin.js
- /Users/nkalinov/dev/simonapp/.yarn/cache/webpack-npm-4.41.5-ec6f53e9d3-1.zip/node_modules/webpack/lib/web/JsonpTemplatePlugin.js
- /Users/nkalinov/dev/simonapp/.yarn/cache/webpack-npm-4.41.5-ec6f53e9d3-1.zip/node_modules/webpack/lib/WebpackOptionsApply.js
- /Users/nkalinov/dev/simonapp/.yarn/cache/webpack-npm-4.41.5-ec6f53e9d3-1.zip/node_modules/webpack/lib/webpack.js
- /Users/nkalinov/dev/simonapp/.yarn/$$virtual/webpack-dev-server-virtual-d46dc61805/0/cache/webpack-dev-server-npm-3.10.1-34b49b6a2b-1.zip/node_modules/webpack-dev-server/bin/webpack-dev-server.js

ERROR in chunk main [entry]
simon-main.edd606415a1133a91f8b.js
Can't open file: No file descriptors available
Require stack:
- /Users/nkalinov/dev/simonapp/.yarn/cache/webpack-npm-4.41.5-ec6f53e9d3-1.zip/node_modules/webpack/lib/web/JsonpMainTemplatePlugin.js
- /Users/nkalinov/dev/simonapp/.yarn/cache/webpack-npm-4.41.5-ec6f53e9d3-1.zip/node_modules/webpack/lib/web/JsonpTemplatePlugin.js
- /Users/nkalinov/dev/simonapp/.yarn/cache/webpack-npm-4.41.5-ec6f53e9d3-1.zip/node_modules/webpack/lib/WebpackOptionsApply.js
- /Users/nkalinov/dev/simonapp/.yarn/cache/webpack-npm-4.41.5-ec6f53e9d3-1.zip/node_modules/webpack/lib/webpack.js
- /Users/nkalinov/dev/simonapp/.yarn/$$virtual/webpack-dev-server-virtual-d46dc61805/0/cache/webpack-dev-server-npm-3.10.1-34b49b6a2b-1.zip/node_modules/webpack-dev-server/bin/webpack-dev-server.js
ℹ ｢wdm｣: Failed to compile.

To Reproduce

The minimal information needed to reproduce your issue (ideally a package.json with a single dep). Note that bugs without minimal reproductions might be closed.

IMPORTANT: We strongly prefer reproductions that use Sherlock. Please check our documentation for more information: https://next.yarnpkg.com/advanced/sherlock

Screenshots

If applicable, add screenshots to help explain your problem.

Environment if relevant (please complete the following information):

• OS: OSX
• Node version: v12.14.1
• Yarn version 2.0.0-rc.28

Additional context

Let me know if webpack conf would be helpful… Any additional tweaks needed to support recompilation/hmr?

Thanks!

[rtsao]

I’m running into this issue with a larger app with roughly 1000 package dependencies.

Merely building the app works fine, but with watch I get the same Error: Can't open file: No file descriptors available.

After some digging, I think I’ve narrowed down the problem to watchpack, and specifically this method:

https://github.com/webpack/watchpack/blob/338675e975e798eabbfd7f99f8991ae0a0c42ad2/lib/watcherManager.js#L16-L32

One hacky workaround I’ve found is:

	getDirectoryWatcher(directory, options) {
		if (DirectoryWatcher === undefined) {
			DirectoryWatcher = require("./DirectoryWatcher");
		}
		options = options || {};
+		const key = directory.includes(".yarn/$$virtual")
+                ? "yarn_virtual" // Shared watcher for virtual packages
+                : directory + " " + JSON.stringify(options);
-		const key = directory + " " + JSON.stringify(options);
		const watcher = this.directoryWatchers.get(key);
		if (watcher === undefined) {
			const newWatcher = new DirectoryWatcher(directory, options);
			this.directoryWatchers.set(key, newWatcher);
			newWatcher.on("closed", () => {
				this.directoryWatchers.delete(key);
			});
			return newWatcher;
		}
		return watcher;
	}

This change prevents the file descriptor error – I assume the watchpack DirectoryWatcher doesn’t play nicely with PnP. I’m not sure if it really even makes sense to have filesystem watchers for virtual packages because they don’t exist on disk and are immutable. So I modified the caching mechanism so there’s only a single DirectoryWatcher for all the virtual packages, which avoids the file descriptor exhaustion problem.

[hydrosquall]

I’m experiencing this same issue while configuring the Sanddance project inside of a react-cosmos project using yarn 2. Will post again if I can make a minimal public repository, but I’m able to trigger this condition every time on a private project.

Update

• I’m able to reproduce the issue by running any application on another port (Jupyter notebook, Python Flask Server, an instance of node-http-proxy). The moment I close another app, the webpack-dev-server is able to recompile successfully.