[Bug] Extremely slow `yarn run xxxx` commands on large monorepo

The bug was explained there by my colleague @etiennedupont , but maybe it should be reported here as it concerns v2 ?

https://github.com/yarnpkg/yarn/issues/7917

Bug description

When running scripts defined in package.json using yarn run xxx on a large monorepo, there is an extremely long delay before the command is actually run. This delay is always the same and seems to depend on the size of the monorepo.

Command

Here is a simple example:

In package.json

{
  "scripts": {
    "echo": "echo ok"
  }
}

Then in shell run:

> yarn echo

.... WAITS FOR 265s .....

ok

Using time to confirm the duration:

> time yarn echo
ok
yarn echo  264.68s user 1.33s system 99% cpu 4:26.01 total

What is the current behavior?

Yarn does something using 100% of a CPU core for 265s (on my repo and machine) before actually running the command.

What is the expected behavior?

Yarn runs the command instantly

Steps to Reproduce

• Have a large monorepo with 176packages in the workspace.
• yarn install
• Run any yarn run xxxx command in any of the packages folder.
• Environment

Node Version: 10.15.3 Yarn v1 Version: 2.0.0-rc.29.git.20200218.926781b7 OS and version: MacOS Catalina

[arcanis]

To give you some context, the way the resolvePeerDependency function works, it traverses the tree to figure out for each node N what are the dependencies it needs to extract from its parent P to satisfy the peer dependencies of N - then once it’s done, it recurses inside the newly created package N2 (which doesn’t have peer dependencies anymore) in order to let its transitive dependencies inherit their peer dependencies as well.

I’m not entirely sure of the circumstances under which you’d end up with a huge tree - there’s likely a mix of peer dependencies and regular dependencies that trigger a catastrophic expansion. My main guess is that something somewhere (possibly in multiple places) is listed as a regular dependency instead of being a peer dependency, causing Yarn to dig into it and generating those huge graphs. If you print parentLocator.name in your console.group you might get a clue by seeing which package is referenced the most.

[arcanis]

Hey @RaynalHugo! I took a quick look at generating a huge monorepo with 200 packages, but no luck - my run time was mostly unaffected (I used the following script inside a project configured with a packages/* glob pattern:).

for x in {1..200} ; do
    (mkdir -p packages/pkg-$x && cd packages/pkg-$x && yarn init)
done

I think the best would be to use --no-minify as you guessed, and put some timing statements around setupWorkspaces and resolveEverything. Depending on the results other places may be good profiling targets as well, such as initializePackageEnvironments.