[Bug?]: Yarn up does not upgrade nested git dependencies

Self-service

• I’d be willing to implement a fix

Describe the bug

Nested git dependencies do not update with yarn up package@org/repo

I don’t know if this is specifically a git problem. If you have git dependencies which themselves have git dependencies, the tree as a whole does not update properly using yarn up and even doing yarn remove + yarn add doesn’t fix the problem. These 2-level-deep dependencies are deadlocked at whatever version they resolved to when first added.

To reproduce

repoA: yarn add mypackage@myorg/repoC

repoB: yarn add mypackage@myorg/repoC

repoX: yarn add mypackage@myorg/repoA yarn add mypackage@myorg/repoB

Now make changes to repoC, and upgrade it in both repoA and repoB.

And there is now no way to ever upgrade repoC in repoX. Upgrading repoA and repoB will forever still resolve repoC to whatever it was before in repoX. Even though both repoA and repoB have the latest in their yarn.lock files. repoX will have a deadlocked repoC.

Environment

yarn dlx -q envinfo --preset jest

  System:
    OS: macOS 11.6
    CPU: (8) arm64 Apple M1
  Binaries:
    Node: 16.8.0 - /private/var/folders/2d/jfjs6qcj04d7skq07hhk3tdw0000gn/T/xfs-b3dac47f/node
    Yarn: 3.1.0 - /private/var/folders/2d/jfjs6qcj04d7skq07hhk3tdw0000gn/T/xfs-b3dac47f/yarn
    npm: 7.21.0 - /opt/homebrew/bin/npm
  npmPackages:
    jest: ^26.6.2 => 26.6.3

Additional context

No response

[em]

If anyone else has this issue, I’m solving by adding separate scripts for all internal dependencies to package.json like so:

scripts: {
  "upgrade-company-lib": "head=$(git ls-remote https://github.com/company/lib.git master | cut -f1) && yarn up company-lib@company/lib#$head"
}

So I don’t have to git log and copy/paste commits all day.

@arcanis if commit IDs are necessary, and as you say recommended, perhaps yarn cli add and up could do this automatically and default to writing the commit ID to the package.json - or even disallow non-deterministic resolutions all together since they create this edge case for deterministic installs.

FTR, this problem created production downtime for us, because our integration was relying on the assumption that upgrading a package would result in the same dependencies that an isolated install of the package had, which was always the case with Yarn1. The isolated package passed tests but failed after integration due to stale transitive dependencies. Further integration testing is a gap in our coverage for sure. But the root problem yarn being nondeterministic. Ironically because of its attempts to be more deterministic as I now understand from your explanation.

[arcanis]

the yarn.lock file in the direct dependency has changed and is itself resolved to a different commit for the transitive dependency.

Transitive lockfiles have no bearing on the resolution (if only because Yarn will always resolve one name/range combination to a single fixed resolution, which would be impossible if two lockfiles were both sources of truth).

the original transitive dependency didn’t have a commit, meaning it not a static reference, and has to always be revalidated.

The resolution turns all ranges into static references, that’s the very reason why we can say that a project install is deterministic: because the lockfile stores what it has installed, and subsequent installs use the same references.

You do upgrade transitive dependencies when the direct dependency specifies a semver range that is incompatible with the current one…

Because it’s not the same range, hence we need to resolve it. There’s a decent case I guess that yes, if a dependency is ^1.0, if your resolution is currently 1.2, if the range changes to ^1.1 after the dependent got upgraded, then Yarn could upgrade it to a newer 1.3 even if 1.2 would still work fine, but in practice I’ve never seen this case happen, or matter (dependency ranges are unlikely to go from ^1.0 to ^1.1 if a 1.2 release exists; it would just rather jump directly to ^1.2).