Nested directory support

I currently access packages like this import { foo } from 'foo/bar';. It seems the current bootstrap only creates node_modules/foo/index.js with module.exports = require('/full/path/packages/foo');. Thus when accessing foo/bar, it Error: Cannot find module.

What would be the best path forward for supporting this? The only thing I can think of is to find all *.js paths and then create that path with matching exports.

[dchambers]

Yay! Great work to everyone that helped to get this issue across the line 👍

[patrickheeney]

This is a summary of some different options discussed above:

Index Proxy - This what lerna looks to be doing right now. It generates a proxy to require the real path of package.json main.

Cons: It only supports one entry file. Nested files does not work.

Proxy Everything - Lerna could be expanded to proxy all files to point to the real path so that require will work for nested paths.

Cons: How would it handle non-js files? What happens when you create new js files (manual step to rebuild the proxy? watchers?).

NPM Link - Adapt lerna to npm link all packages.

Cons: Complex with lots of packages. Can only have one version of a dependency linked at a time. Other issues documented above and elsewhere.

Symlink - Create a symlink process to link all the modules. Not sure this is possible on its own without the next point.

Cons: Will it work on windows smoothly? How to avoid complexity? Was mentioned above it had some issues with recursive node modules.

Global Cache / Symlink - Using tools like pnpm or other projects that store all npm packages in a global cache. Then recursively symlink to link them.

Cons: Will it work on windows smoothly? Adds another dependency. Can’t use npm directly for a few commands.

Patch NPM - Patch the NPM require process to change its behavior.

Cons: NPM won’t change how modules will load due to standardization. Patching the require process will be brittle and not future proof.

---

I have done some extensive testing of all of these. My current thoughts:

The proxy everything approach is the easiest to roll into the project now. It is also error prone because of the inability to add new files, change paths, etc without a manual step to rebuild or initiate watchers. It seems like a perfect starting point.

The global cache / symlink approach I have been using outside of learna, but I’d like to roll this into it. A fork also exists to do something similar. I personally love this approach as it is what I think npm 3 should have been. However it does depart a bit from npm’s default behavior. It is the best candidate that I have found to handle this seamlessly.

The others all have drawbacks that prevent this from likely working seamlessly.

One interesting thought is, can we use both approaches behind flags? Use proxy everything by default and enable the global cache / symlink with a flag. We are essentially just talking about two strategies to proxy the node_modules, so it may be doable. See this example.

lerna bootstrap --strategy=copy

Another interesting thought, if this isn’t bundled into core because of additional dependencies, can it be a plugin? By default the core project can only proxy everything but you can include a lerna-global-cache plugin that implements this strategy. There is a definite need for something like this and this could open it up to the community for other strategies instead of depending on something like pnpm directly. lerna shouldn’t concern itself with how the modules are bootstraped, it just expects it to behave like npm does.

// lerna requires() some root project level function to modify configuration
lerna.__commands__['bootstrap'] = class GlobalCacheBootstrapCommand extends Command {}

I know it was said in other issues that lerna doesn’t want different types of structures between repos and wants to focus on one way of doing this. However, focusing on only one use case will not work for everyone. So the question becomes do you support multiple strategies out of the box, allow plugins to modify this behavior instead of forking, or reject this and similar use cases.

EDIT: Added last two paragraphs.