strategies for installing dependencies of local dependencies

Scenario

• parent-pkg has local dependency: file: ./local-pkg.
• npm install in parent-pkg.
• ./local-pkg is copied and installed into parent_package/node_modules, along with the dependencies oflocal-pkg`.
• post-install, run linklocal.
• node_modules/local-pkg is now replaced with a relative symlink to ../local-pkg. Dependencies of local-pkg are not installed.
• Use linklocal list to produce a list of all local dependencies, which we then pass to something like xargs, find -exec or bulk to run npm install --production for us: linklocal list | bulk -c "npm install --production".
• Local dependencies have now got their dependencies installed.

Issues

1. Dependencies of local dependencies were installed twice. First when the local dependencies were installed with the regular npm install, second during the bulk npm install in the local dependencies. This can lead to considerable increases in install time if you depend on anything that requires building.
2. No deduplication occurs between local dependencies. If two local packages depend on the same thing, it will be installed twice.

Some Improvements

Don’t hit remote registry

Disable accessing the remote registry with npm install --cache-min=Infinity for the bulk install. You’ve already just installed everything you need so there’s a good chance everything is already in the cache. This prevents overhead of hitting npm servers over and over for every local dependency. Doesn’t solve rebuilding over and over again though.

Don’t install local dependencies then symlink them away

To avoid the initial install of everything before symlinking it all away, I’ve been doing linklocal in the preinstall script. Of course this requires linklocal and bulk to be installed before starting, so my preinstall (Makefile) looks something like:

preinstall:
    npm install linklocal bulk # this unfortunately ignores the version specified in package.json
    linklocal link -r
    linklocal list -r --unique | bulk -c "npm install --cache-min=Infinity"

but what’s really killing install time is rebuilding dependencies over and over due to lack of deduplication.

cc @hughsk @yoshuawuyts any ideas on what we can do to improve this?

[b-smets]

Any progress on this issue yet? The install time due to the repeated rebuilding of dependencies is really bad. I don’t think @jbach 's approach works with npm3 as the local package will not have its own node_modules folder when installed (before the linking). You would have to manually resolve all dependencies (and their dependencies and so on) and cherry pick them all into the local package module.

[jbach]

Old issue, but still: I think the “stealing” approach is the way to go. Considering npm3’s behaviour, node_modules/local-pkg/node_modules should only include module versions that are specific to local-pkg, thus not in the flat hierarch. Copying the node_modules/local-pkg/node_modules folder to ./local-pkg/node_modules before symlinking works, at least in npm3. I’m currently doing exactly that by script but would love to replace it with linklocal.