[Bug] Link step when installing on a moderately sized monorepo is taking a very long time

• I’d be willing to implement a fix

Describe the bug

Yarn takes upwards of 9 minutes in the link step phase of a monorepo install. In one case it took 39 minutes.

During this time no logs or other useful information is output.

Related to the problems @haggholm had in #818

To Reproduce I am sure this is going to be a pain to reproduce however in my case:

• Have a monorepo with 19 different packages, some depending on each other in their peerDeps and devDeps
• Run yarn

I am attempting to create a non-private monorepo where this can be reproduced on.

Screenshots

Environment if relevant (please complete the following information):

• OS: OSX
• Node version: 11.0.1
• Yarn version: Running from source

Additional context

I found the traverse method found on this line frequently is taking a long amount of time:

https://github.com/yarnpkg/berry/blob/8b8483bc95f3e39e597c77087e61dc2c2fc21447/packages/yarnpkg-core/sources/Project.ts#L1141

It seemed to hang when processing a package that had a dependency on a common package in my monorepo.

Changing traverse(resolution, new Set(seenPackages)); to traverse(resolution, seenPackages); speeds everything up, taking only 46seconds, but of course this is fundamentally changing what this method is doing. I don’t have enough knowledge about what exactly this is trying to do to try and fix it myself.

Feel free to reach out for further information though or other ways of debugging what’s going wrong here. I’ll also try and create a reproduction repository.

[arcanis]

Even better, instead of computing the build hash through digest(A, B, C) (assuming that A depends on B which depends on C), we should instead compute it via digest(A, digest(B, digest(C))). This would allow us to cache in memory the intermediary results, so we’d never need to traverse the same package twice.

[arcanis]

I think this should be fixed with the latest improvements in master! (cf #1040) I’ll go ahead and close it, we can open another issue if there’s something else to do.