Pipe Dream: Coursier "Solve"

What follows is part brain-dump, part RFC, and part “somebody please do this because dear god I need it but I don’t have enough time”. Happy for this to exist somewhere else if the maintainers prefer, but I thought it made sense here.

Problem

You depend on a set of libraries. Let’s say the following:

• Http4s
• Redis4Cats
• Monix Catnap
• Ciris
• Pick any two of the fifty things in the Davenportverse
• Fs2

(to be clear, these are your direct, stated dependencies in your build.sbt)

You want to upgrade any one of these. For example, assume http4s. However, upgrading one results in the other now depending on an old version of something transitive. Maybe catnap is now broken because http4s pulls in a newer version of Cats Effect, or conversely maybe there’s a better version of Cats Effect that can be grabbed and explicitly evicted to. Upgrading http4s potentially messes up downstream things, such as almost anything in the Davenportverse, so that’s a concern.

It’s very complicated and hard, and ultimately requires a relatively expert-level knowledge of the ecosystem and its transitive dependencies (and who happens to be maintaining bincompat and who isn’t!) in order to figure it all out. It’s also very time consuming to do by hand.

To be clear, this problem generalizes to any polyrepo distribution system. Companies which use polyrepos actually feel this problem far more acutely than the public ecosystem (believe me…). This, in a nutshell, represents the strongest objective argument against an extensible ecosystem: it’s incredibly difficult to identify “compatible sets”.

Possible Solution

What if we could just… ask Coursier? Think about something like this (literally making up syntax here):

$ coursier solve --scala 2.13 org.http4s::http4s-server org.typelevel::cats-effect:2.2.0 dev.profunktor::redis4cats-effects 'co.fs2::fs2-io:2.[3,)'

Imagine if Coursier then would spit out the most recent, maximally-compatible set of dependencies, ideally in a way that can be copy/pasted into a libraryDependencies declaration. The idea here is that we’re expressing a set of constraints (note the missing versions). We want these things, some of which with this specific version, some with an Ivy range, and some without any version at all, and we want the solver to figure out what our ideal build configuration should be in order to ensure everything is mutually compatible but also upgraded as far as possible.

Additionally, note the Scala version is explicitly included here. This is partially for convenience (so we can use the fictional :: notation), but also so that Coursier can find versions which comply with our version. Not everyone is on the latest Scala, and sorting out compatible sets on older versions while upgrading as much as possible is a massively non-trivial problem, particularly when libraries like Circe have conditional dependencies that jump between breaking lines and other non-linear things.

Obviously this isn’t possible right now. But I think it could be.

Implementation

There are a couple things that would be needed for this. The most obvious one is some assumptions about declared compatibility in versioning. Coursier already defaults to Ivy’s eviction rules on this one, and I think that’s fair, but it should be overridable on a per-artifact basis. For example, Cats Effect is fully binary compatible within its major lines, and additionally happens to be binary compatible between 1.x and 2.x (Cats is as well), and it should be possible to declare this somewhere in a fashion that all users have access to it. Ideally this would just be in the POMs, but we can’t necessarily do that, so some external metadata mechanism is probably necessary. This wouldn’t need to be in MVP, but it would be helpful.

The biggest piece though is the solver itself. From what I remember from the last time I looked into this, Coursier’s resolver is almost powerful enough to do this, save for one critical piece: it doesn’t support backtracking. Coursier’s resolver isn’t a general constraint solver, it’s just a straightforward iteration algorithm that attempts to resolve conflicts by evicting forward. This is entirely sufficient to replace Ivy in sbt, but it’s not enough to achieve this idea.

However, I don’t think it would be hard to make it enough. Constraint solvers honestly aren’t that hard to write (stick your constraints into a set, then iterate on that set until you either instantiate every variable or you complete an iteration without making any changes), and implementing this would greatly extend Coursier’s capabilities and allow it to fill this critical and glaring need within the ecosystem.

[keynmol]

I’ve actually started on a project which has a similar goal - a view of the ecosystem as a multi-graph, that can be traversed from a set of requirements. In my case the requirement was “is it ready for 2.13?” which is reflected in the labeling (green/red).

The original idea was to guide people who want to contribute to Scala version upgrades in the ecosystem that are desperately needed, but this has similar problems - the quality of the raw data that can be sourced from, say, Scaladex, is not good enough to solve for “Pick a subgraph that will be completely in 2.13 if you upgrade this minimal set of dependencies in this order”

(all the red stuff in the corner is Scalameta’s seemingly abandoned projects 😃

[eed3si9n]

Coursier already defaults to Ivy’s eviction rules on this one, and I think that’s fair, but it should be overridable on a per-artifact basis.

sbt 1.4.0 will add ThisBuild / versionScheme (https://github.com/sbt/sbt/issues/5710), and there’s a plugin called sbt-version-policy that Alex wrote that lets you locally override similar information:

versionPolicyDependencyRules += "org.scala-lang" % "scala-compiler" % "strict"

The biggest piece though is the solver itself.

When we ask a module with a few libraryDependencies effectively it’s doing the solving. What’s NOT happening (and it should) is resolver saying “I’m sorry I can’t” when the constraints are incompatible. So I think what we are asking is strict mode based on more accurate compatibility check based on Semantic Versioning, PVP, etc declared by the library.