Add explicit resolution capabilities to bsp

buildTarget/dependencySources is one of the most important requests in BSP. dependencySources allows language servers to manage dependencies’ indices and provide language features like syntax highlighting, code navigation and autocompletions. These features are supported for both the projects of a build and the library dependencies (when you go to the definition of a method in a library, you want to be able to go to definitions of its body).

To achieve this full experience, IDEs extract all source/doc jars of the build via their custom integrations (think sbt, maven or gradle integrations in IntelliJ) and index the source code. This is a slow process that makes the “import project” experience poor. When developers clone projects and import them, the process can take easily more than 10 minutes because the build tool needs to download all jars (whose size can be in the order of GB for a medium-sized Scala project).

All Scala developers that want to contribute to a library for the first time need to go through this expensive process. I’ve noticed that in my case I’ve tried hard to avoid cloning projects locally and preferred making changes via the github interface to avoid the wait, but ultimately made the contributing experience even slower without the help of a local IDE to compile and browse.

A better solution

The problem here is that the download of jars (binary, source and docs) happens upfront when the user clicks “import project” in the IDE. In most of the cases, the download of source and doc jars are not necessary because users will most likely never browse all dependencies of a project!

I think we could address this problem by making the source and doc jar resolution lazy and letting the language server handle it. dependencySources would be free not to return any source jars, and IntelliJ, metals or Dotty IDE would ask the build tool to resolve the jar of a dependency when the user opens up a file that hasn’t been indexed.

This approach will save disk space in people’s computers and save time when doing the full import project in the IDE.

This enhancement would require the addition of a ResolutionProvider and a buildTarget/resolve endpoint (similar to `buildTarget/dependencySources). This endpoint would take a list of URIs (binary jars) and return a list of (resolved) source jars. The idea is that the build tool will only resolve the source jars for the requested binary jars. We may want to make some parts of this language-specific.

/cc @jastice

[olafurpg]

I see what you mean. I agree this might be a good idea, and one example of how client/server may be better than eagerly persisting all information to JSON/XML on disk. I would let it wait for a future version of BSP (maybe v2), I suspect it requires custom work on both sides (IDEs + build tools) to make this feature work well.

[jastice]

@retronym technically yes, there even is an existing mechanism for first pass/second pass import, but we effectively skip the first pass so that the user can start editing even if the server fails completely. Adding a third pass to this might be more complicated though, and would be more resource-consuming. Partial updates of imports and attaching sources later is also possible, but makes things more complicated, too.