Dealing with version clashes on transitive dependencies

I work on a Monorepo that contains several Java, Scala and Kotlin projects, all within a single WORKSPACE. Using bazel-deps worked well for a while, but as the repository grew, it has become hard to deal with transitive dependencies version clashes.

To address that, I attempted to use a dependencies.yml for each project, but it didn’t work out – the order in which we imported each project/repository on the WORKSPACE was considered when resolving conflicts. Did I do something wrong, or is it working as intended?

Is it possible to have several versions of a single dependency on the dependencies.yml file, and force which version we want for each Bazel target? It looks like rules_jvm_external supports this.

Creating WORKSPACE files for each project is looking like the way to go, but we wanted to avoid that as much as possible.

[ianoc-stripe]

Once you have gotten into the state its pretty hard to get out of. Our main internal control in the area’s I care about is that we heavily restrict the PR’s that can touch the workspace.bzl file, and i basically read the conflicts when it changes to decide if it seems ok or not. Having a version conflict of failed is probably pretty good for most things, some stuff like finagle, or guava can be a bit of a wild card even in this.

I would probably try the fail/fixed first to see how you can manage it, but overall i think using that or not this is going to boil down to probably visually deciding about transitive deps stability/bumping things + requesting new publishes or dropping the use of external libraries that aren’t updated/need older versions of things. Its all pretty hassle filled in any big monorepo i’ve worked on, having the reviews or settings can mostly stop it getting into the repo, but the more correct you are in what goes into your repo can mean the more effort is required to do an upgrade (which is the only stable way to do the upgrade tbc, but its time consuming, e.g: Upgrade Lib A depends on newer C which then means you need to bump B, which has a knock on effect on libraries E,F,G… etc… It does really pay to have a shallow dependency tree and I would shade something like spark where you can to use that to slim down the dependencies there)

[pbsf]

Thanks for the input. The runtime issues you guys mentioned are the ones we are dealing with atm.

We already use bazel_jar_jar on a few cases. Might be the way to go indeed. The bigger problem is that our dependencies.yml has 80 conflicting dependencies, with versionConflictPolicy=highest. It is now too hard to fix these on a case-by-case basis. Most of these conflicts are not causing runtime errors.

We are thinking about creating a new dependencies.yml file with versionConflictPolicy=fail/fixed, load it earlier than the legacy one on the WORKSPACE, and fix the conflicts as it arises on the new file. Does that seem like a reasonable migration approach? Would you recommend fail or fixed as the conflict policy?