sbt resolves dependencies differently with/without coursier

Hi,

Since sbt switched to Coursier I’ve faced with the issue described in https://github.com/sbt/sbt/issues/5389. It looks like the root cause of the issue is a different classpath we get using Ivy and Coursier.

So, here is a problem repo https://github.com/rusexpertiza-llc/yupana. The subproject is hbase. Here is the diff between Ivy dependencies and Coursier:

- guava-12.0.1.jar
+ guava-16.0.1.jar
+ hadoop-mapreduce-client-core-2.5.1.jar
+ hadoop-yarn-api-2.5.1.jar
+ hadoop-yarn-common-2.5.1.jar
+ jline-0.9.94.jar
+ netty-3.7.0.Final.jar
- protobuf-java-2.6.1.jar
+ protobuf-java-3.8.0.jar
+servlet-api-2.5-20081211.jar

One of the problems I see here is that forced version for protobuf-java is ignored.

[eed3si9n]

It’s interesting that the dependency resolution differences seem to be around Hadoop related things. For example, if you look at cs resolve org.apache.hadoop:hadoop-common:2.9.2 -t, you’d see the libraries that are mentioned here like netty-3.7.0.Final and nimbus-jose-jw.

It would be good to compare not just Ivy vs Coursier, but also the resolution result of Maven as well because some of the differences could be coming from the fact that Coursier is doing a better job emulating Maven, especially when it comes to traversing the parent POMs.

Hadoop depends on Apache Curator Framework. https://repo1.maven.org/maven2/org/apache/curator/curator-framework/2.7.1/curator-framework-2.7.1.pom At a glance it doesn’t look like it doesn’t depend on much:

<dependencies>
  <dependency>
    <groupId>org.apache.curator</groupId>
    <artifactId>curator-client</artifactId>
  </dependency>
  <dependency>
    <groupId>org.apache.curator</groupId>
    <artifactId>curator-test</artifactId>
    <scope>test</scope>
  </dependency>
</dependencies>

but wait there’s more…

<parent>
  <groupId>org.apache.curator</groupId>
  <artifactId>apache-curator</artifactId>
  <version>2.7.1</version>
</parent>

Here’s the parent POM - https://repo1.maven.org/maven2/org/apache/curator/apache-curator/2.7.1/apache-curator-2.7.1.pom

<dependencies>
  <dependency>
    <groupId>org.apache.zookeeper</groupId>
    <artifactId>zookeeper</artifactId>
  </dependency>
  <dependency>
    <groupId>com.google.guava</groupId>
    <artifactId>guava</artifactId>
  </dependency>
  <dependency>
    <groupId>org.testng</groupId>
    <artifactId>testng</artifactId>
    <scope>test</scope>
  </dependency>
  <dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-log4j12</artifactId>
    <scope>test</scope>
  </dependency>
</dependencies>

That’s Guava 16 and Zookeeper 3.4.6. Hadoop also depends on Zookeeper but with excludes. Curator doesn’t have those excludes.

What does Zookeeper 3.4.6 depend on? https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.pom

<dependency>
  <groupId>jline</groupId>
  <artifactId>jline</artifactId>
  <version>0.9.94</version>
  <scope>compile</scope>
</dependency>
<dependency>
  <groupId>io.netty</groupId>
  <artifactId>netty</artifactId>
  <version>3.7.0.Final</version>
  <scope>compile</scope>
</dependency>

That’s right JLine 0.9.94 and Netty 3.7.0.Final.

I mean you probably don’t need those (and that’s why Hadoop excludes them), but when I see extra JLine or Netty, I usually assume that Coursier is being correct.

[bpossolo]

upgraded from sbt 1.3.4 to 1.4.7 (required because im on a new M1 MacBook Air)… I’ve noticed major differences between the coursier and ivy classpaths as well…

for example my project depends on com.microsoft.azure:msal4j:1.8.0 that lib depends on com.nimbusds:oauth2-oidc-sdk:7.4 which depends on com.nimbusds:nimbus-jose-jwt:8.14.1

coursier doesn’t even download either of those two jars nor add them to the classpath…

I had to disable coursier to get my project to function. ThisBuild / useCoursier := false

seems odd to rewrite dependency management when maven/ivy are tried and trued…