[Bug?]: Yarn downloads irrelevant optional dependencies (e.g. for a different OS)

Self-service

• I’d be willing to implement a fix

Describe the bug

I’m the main developer behind esbuild and I’m working on changing the installation strategy for the binary executable from an install script to a list of optional dependencies, one for each supported OS and architecture combination. This works fine with npm and pnpm but doesn’t work well with Yarn because Yarn always downloads all optional dependencies instead of only downloading the optional dependency for the current OS and architecture. So switching installation strategies would presumably make installing esbuild with Yarn a lot slower than it should be.

The motivation for doing this is to move the complexity of figuring out what to download from where out of the esbuild package and into the package manager instead. That way installing esbuild should “just work” in complex scenarios including when offline, with a custom registry, with a custom proxy, or with limited file system permissions. Some of these scenarios don’t work with esbuild’s current install script and it’s sometimes very hard to debug these issues since it’s hard to replicate a user’s network environment. See https://github.com/evanw/esbuild/issues/789#issuecomment-901467782 and the surrounding thread for more context.

To reproduce

These specific instructions are for macOS:

$ echo '{ "dependencies": { "esbuild-experimental-optdeps": "0.12.3" } }' > package.json
$ touch yarn.lock
$ rm -f ~/.yarn/berry/cache/esbuild-*
$ yarn set version berry
$ yarn
$ ls ~/.yarn/berry/cache/esbuild-*
esbuild-darwin-64-npm-0.12.3-3bce268eff-8.zip
esbuild-darwin-arm64-npm-0.12.3-2067324df1-8.zip
esbuild-experimental-optdeps-npm-0.12.3-da58fe24de-8.zip
esbuild-freebsd-64-npm-0.12.3-b28b30bfa6-8.zip
esbuild-freebsd-arm64-npm-0.12.3-5601b669f5-8.zip
esbuild-linux-32-npm-0.12.3-182249b5a7-8.zip
esbuild-linux-64-npm-0.12.3-4067979d42-8.zip
esbuild-linux-arm-npm-0.12.3-91320e7a3b-8.zip
esbuild-linux-arm64-npm-0.12.3-ef73807905-8.zip
esbuild-linux-mips64le-npm-0.12.3-daf3dd1009-8.zip
esbuild-linux-ppc64le-npm-0.12.3-81866b1f7b-8.zip
esbuild-windows-32-npm-0.12.3-778762815e-8.zip
esbuild-windows-64-npm-0.12.3-5ac80b68d1-8.zip

The esbuild-experimental-optdeps package is one I have made specifically to test this optional dependency installation strategy. As you can see here, all 13 packages have been downloaded instead of only downloading the 2 relevant packages like both npm and pnpm. See https://github.com/evanw/esbuild/issues/789#issuecomment-901467782 for more detail about npm and pnpm.

Environment

System:
  OS: macOS 11.5.1
  CPU: (12) x64 Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
Binaries:
  Node: 16.5.0 - /private/var/folders/j6/np400cw17sz0n5ljd67byrzw0000gn/T/xfs-0b02617d/node
  Yarn: 3.0.1 - /private/var/folders/j6/np400cw17sz0n5ljd67byrzw0000gn/T/xfs-0b02617d/yarn
  npm: 7.20.6 - ~/.npm-local/bin/npm

Additional context

Yarn’s inefficient behavior here isn’t a deal-breaker for me since it isn’t a correctness issue, just a performance issue. But at some point I may move forward with this different package installation strategy for esbuild, at which point installing esbuild with Yarn will become much slower than with npm and pnpm.

[merceyz]

I edited it away as it will unpack all the versions when only one of them will be used, what you’re doing now is more efficient disk usage. It would actually be preferable if you set preferUnplugged: false on them so they wont be unpacked unnecessarily

[evanw]

Heads up that esbuild’s package installation now works this way as of version 0.13.0.