[Bug] allow node_modules to be a symbolic link

• I’d be willing to implement a fix

Describe the bug

For every other directory that yarn wants to write, a symbolic link to a location with write access if sufficient, except for node_modules. Please allow node_modules to be a symbolic link and only check write permissions of its target.

$ yarnpkg add pretty-ms
➤ YN0000: ┌ Resolution step
➤ YN0000: └ Completed in 0s 601ms
➤ YN0000: ┌ Fetch step
➤ YN0013: │ parse-ms@npm:2.1.0 can't be found in the cache and will be fetched from
➤ YN0013: │ pretty-ms@npm:7.0.1 can't be found in the cache and will be fetched fro
➤ YN0000: └ Completed
➤ YN0000: ┌ Link step
➤ YN0001: │ Error: EACCES: permission denied, unlink '/tmp/project1/node_modules'
➤ YN0000: └ Completed
➤ YN0000: Failed with errors in 0s 901ms

To Reproduce

The minimal information needed to reproduce your issue (ideally a package.json with a single dep). Note that bugs without minimal reproductions will be closed as non-actionable.

Create project directories as root user and give ownership of files for which yarn needs write access to a normal user (gitlab in this example),

cd /tmp
mkdir projects
cd projects
mkdir .yarn .node_modules
chown gitlab: .yarn .node_modules .
mkdir project1
cd project1
ln -s ../.yarn .
ln -s ../.node_modules node_modules
ln -s ../.yarnrc.yml .yarnrc.yml
ln -s ../package.json package.json
ln -s ../README.md README.md
ln -s ../yarn.lock yarn.lock
ln -s ../.gitattributes .gitattributes
ln -s ../.gitignore .gitignore
ln -s ../.editorconfig .editorconfig

Now switch to gitlab user and create a project, (note: due to a conflict, yarn command is yarnpkg in debian)

su -s /bin/bash - gitlab
cd /tmp/projects/project1/
yarnpkg init -y
touch .yarnrc.yml 
if ! grep nodeLinker .yarnrc.yml >/dev/null; then echo "nodeLinker: \"node-modules\"" >>.yarnrc.yml; fi
cat .yarnrc.yml 
# prints: nodeLinker: "node-modules"
yarnpkg add pretty-ms

Now it fails trying to remove node_modules. When node-modules is chosen as nodeLinker, it should not try to remove the directory, the user has the required permissions inside the directory pointed by the symbolic link.

This setup is required to follow File System Hierarchy Standard (FHS) in debian, where we want to separate directories where programs are allowed to write and prohibited to write.

So the application files are installed in /usr/share/gitlab and any files or directories that gitlab needs write access is provided as symbolic links to /var/lib/gitlab.

This setup was working well with yarn 1.x, the permission issue started after trying to switch to yarn 2 with node-modules plugin.

IMPORTANT: We strongly prefer reproductions that use Sherlock. Please check our documentation for more information: https://yarnpkg.com/advanced/sherlock

Reproduction

// Sherlock reproduction. For instance:
await packageJsonAndInstall({
  dependencies: {
    [`packageName`]: `x.y.z`,
  },
});

Screenshots

If applicable, add screenshots to help explain your problem.

Environment if relevant (please complete the following information):

• OS: [e.g. OSX, Linux, Windows, …] Debian GNU/Linux unstable/sid
• Node version [e.g. 8.15.0, 10.15.1, …] v12.19.0
• Yarn version [e.g. 2.4.0, …] 2.4.0

Additional context

I’m maintaining gitlab package in debian and I have to use yarn to install node modules not yet packaged in debian. Due to FHS, I cannot provide direct write access to node_modules directory and only a symbolic link with write access at the target location can be provided.

[larixer]

@joker-777 @merceyz This issue was initially in nm linker and was fixed by #2487, but at the time of release of 3.2.0 the pnpm linker cleanup code introduced another issue that affected this use case, which in turn was fixed in: #4206 and merged into master, it is not released yet at the moment. To use Yarn from sources from master one can use yarn set version from sources

[pravi]

1. gitlab is a ruby on rails application which has a lot of nodejs dependencies used in frontend part. You can see the source code at https://gitlab.com/gitlab-org/gitlab-foss
2. When a user runs apt install gitlab, it downloads gitlab.deb which has the source code for gitlab application (ruby + node). It includes Gemfile for ruby dependencies and package.json for node dependencies. All ruby dependencies are packaged in debian, so the ruby dependencies are pulled directly from debian packages by apt. The node situation is more complex, it has 1700+ dependencies and about 1500+ of them are packaged. So the modules that are already packaged should be taken from debian and only the remaining modules should be pulled via yarn install. Earlier I used to remove those modules from package.json and yarn will install the remaining modules. This caused some issues at times when incompatible transitive dependencies are installed by yarn (for example schema-utils, loader-utils and mkdirp, we updated them in debian before their original projects updated). So to mitigate this, I want yarn to be aware of these dependencies and link/portal protocol options makes that possible. See https://salsa.debian.org/ruby-team/gitlab/-/blob/master/debian/patches/0740-use-packaged-modules.patch#L97 which has both approaches right now. I’m gradually moving all packaged dependencies to link/portal dependencies. yarn install is run in postinstall script of gitlab package.
3. Your restatement about node_modules folder situation is correct. All architecture independent code installed by the package should be in /usr/share/gitlab but any files created during installation or by application when it is running should be in /var/lib/gitlab (log files should be in /var/log/gitlab and temporary files like sockets in /run/gitlab).
4. A package in debian main should only include source code and many times the node modules include generated files (output of babel, rollup, terser, typescript etc). Also we want to make sure only a single version of a node module is maintained in debian as much as possible (in extreme cases we do provide multiple versions, but we would like to avoid it). If you take the example of yarn itself, it depends on babel-runtime module from babel 6, but we already moved every other package to babel 7 and we wanted yarn also to use @babel/runtime. This process is challenging and the yarn porting was more challenging than usual (yarn was the last package we ported to babel 7 and blocked removing babel 6 from debian). See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=960120 and https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=972952 We call this process “transition” and we test every package depending or build depending a package before updating a semver major version. We keep both versions in the archive during the transition and once all packages are ported to the new version, we remove the old version. See https://wiki.debian.org/Javascript/Nodejs/Transitions/PostCSS8 for how we are currently updating postcss to version 8. We usually send the patches to corresponding upstream projects and everyone benefits from this process. Though this is really against the established workflow of most projects who don’t mind multiple versions of a module. This insistence on a single version makes security updates very easy as fixing one module in the system means every application is safe from a vulnerability now. But it does give us extra work of porting all applications during library updates. But we feel the long term maintenance is worth the effort of transitions.
5. gitlab has a dependency on yarnpkg and apt install gitlab will also install yarnpkg. https://salsa.debian.org/ruby-team/gitlab/-/blob/master/debian/control#L253 I run yarnpkg set version berry in postinstall script to update yarn to version 2. See https://salsa.debian.org/ruby-team/gitlab/-/blob/master/debian/rake-tasks.sh#L39 I hope you will reconsider not providing native packages for yarn so that it makes it easy for us to ship yarn directly in main. This installation in postinstall script is only allowed for packages in contrib section and not main. Since we don’t have all node modules packaged gitlab is currently in contrib, but eventually we want to move it back to main after packaging all node modules. If you stick to the current method of only shipping yarn 2 via yarn 1, I request you to at least update the dependencies/build dependencies so we can maintain yarn 1 in main better. We can send pull requests if you are open to merging our patches in yarn 1 branch.
6. yarnpkg alias problem can be solved by working with 3 people (cmdtest and its reverse build dependencies maintainers). I have opened a bug for that here https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=977709 this situation is a bit unfortunate, but if these three people agrees, we will be able to change it.
7. yes, I already implemented this work around here https://salsa.debian.org/ruby-team/gitlab/-/commit/0b96d9e2bcf036f48edfab6321a5186db2023b55