Fully-featured Java dependency management

Here’s a proposed scheme for handling 3rd-party dependencies in Buck, starting from the perspective of Java dependency management. The main goal is a complete, sustainable solution for the dependency management needs of Buck Java projects.

Looking for feedback on the tradeoffs involved. CC @Coneko @shs96c @mikekap @davido @bolinfest @bestander.

Java dependency management requirements

Java code that’s built with Buck should use the exact same version of every third-party dependency. So most of the time, dependency management is irrelevant. However it becomes critical when importing third-party dependencies into the source tree.

For Buck projects, here are some Java dependency management requirements observed in practice, when importing libraries from Maven repos:

• Resolution: Given a list of direct dependencies, first resolve transitive dependencies, then write corresponding prebuilt_jar rules and inter-dependencies into the source tree at third-party/java/**/BUCK
• Optionals: For example, a Buck rule for a direct dependency should never transitively pull in slf4j-simple, log4j-over-slf4j, and other “logging implementation” JARs. Only the final java_binary should choose what to use.
• Relocations: For example, org.apache.commons:commons-io has been relocated to commons-io:commons-io. Everything should still work as expected when relocations come into play.
• Banned Dependencies: Assert that certain known-to-be-bad dependencies haven’t been pulled in directly nor transitively.
• Banned SNAPSHOTs: Assert that -SNAPSHOT dependencies (which are mutable) haven’t been pulled in directly nor transitively.
• Require Sane Versions: Ensure that if a dependency appears multiple times in the graph, the latest version of that dependency is imported. Possibly, also assert that the versions appearing throughout the graph don’t span multiple major-version numbers.
• Source JARs: Sources for all transitive dependencies must be available in your IDE so you can click into the source code during coding and debugging.
• Exclusions: In rare cases where something we want to depend on declares faulty or unnecessary dependencies, we use exclusions as an escape hatch. Exclusions are almost never sane, but sometimes pragmatically helpful as a stopgap or a last resort.
• Duplicate Classes: Assert that there are no duplicate classes in all libraries that are transitively used by the project.
• Allow remote_file instead of prebuilt_jar: While prebuilt_jar should likely be the default way to import libraries, in some cases, remote_file (with download.in_build = true in the Buck config) is preferable in context.
• Documentation and Usability: The mental model for “how dependencies work” needs to be thoroughly documented, along with “how to get things done” in practical situations. We also need tools to visualize what happens during dependency resolution.

Proposal: Reuse package management

The dependency management requirements, above, lead to a choice:

1. Reuse existing package management and mitigate known problems.
2. Reinvent package management and solve known problems.

I propose reusing existing package management:

• Main downside: a second tool (e.g. mvnw or gradlew) is needed for dependencies
• Main upside: existing and future work is offloaded so we won’t have to deal with it

When choosing to reuse existing package management, our only task is to define and document a sane mapping from the package manager’s model onto Buck’s model. The upside becomes compelling, I think, when evaluating this decision as a potential general pattern to apply to Buck integration of all package managers across all languages.

The following section vaguely sketches what user-facing docs might look like when taking the “reuse” approach with Maven as the package manager. (Maven’s just an example — I expect the general approach to work similarly with Gradle or Ivy.)

Example Docs: Importing dependencies via Maven POM

Run these commands to get a template POM whose contents can be edited to specify your dependencies. The template POM’s comments describe how each section in the POM affects the Buck rules that’ll be generated.

$ cd "$(buck root)"
$ mkdir -p third-party/java
$ cd third-party/java
$ mvn archetype:generate                  \
  -DarchetypeGroupId=com.buckbuild        \
  -DarchetypeArtifactId=third-party-java  \
  -DarchetypeVersion=1.0                  \
  -DgroupId=com.example.yourGroupId

Modify the resulting pom.xml in your editor or IDE. For example, to add the Guava library:

<project>
  ...
  <dependencies>
    <dependency>
      <groupId>com.google.guava</groupId>
      <artifactId>guava</artifactId>
      <version>19.0</version>
    </dependency>
    ...
  </dependencies>
  ...
</project>

Every time the POM is edited, an explicit command must be run to regenerate the third-party/java build files.

$ cd third-party/java
$ mvn verify buck:regenerate-build-files

The resulting file at third-party/java/com.google.guava/BUCK will make the Guava library available to your project. Elsewhere in your project, you can depend on Guava like this:

java_binary(
  name = 'example-rule-that-depends-on-guava',
  deps = [
    '//third-party/java/com.google.guava:guava',
  ],
)

Next steps

I’ve investigated this issue enough that I understand how Maven’s package management can be sanely mapped to Buck rules to meet the requirements described above, but I didn’t describe that mapping here because first we need a higher-level discussion about the tradeoffs in the overall approach.

After understanding the tradeoffs in the high-level approach, the next step is to refine and code the logic needed to fully meet Buck’s Java dependency management requirements.

[davido]

There is no need to actually define a build.gradle file to take advantage of gradle for this. The artifact rule you described above can be translated internally to a model gradle/maven understand and use their core logic to do the rest.

I agree. I wouldn’t care what Buck is using behind the scenes, as far as I don’t have to interact with any specifics of third party tool chains. Say edit (or even see) pom.xml and build.gradle.

except handling transitive dependencies

I think this makes it not very useful as it is almost similar to a vanilla buck fetch with some scripting to figure out where the jar/aar lives on maven central.

Yes. That why I said, that dependency management should be improved in Buck itself.

[mikekap]

Sounds like a great approach. In case you haven’t seen it, buck has a maven importer already that can import pom.xml files (though probably not in a 100% foolproof way). You can run it via buck run maven-importer in the buck repo. The source code is at https://github.com/facebook/buck/blob/master/src/com/facebook/buck/maven/Resolver.java . You may want to mention how far off that is from what you’d like to see.