[Question] - Retrieving sbt plugin versions from hosted Sonatype Nexus

At my work we use Scala Steward to update all of our internal dependencies. We use a hosted Sonatype Nexus to both have a mirror of Maven Central and also to have our own internal releases Maven repo. From looking through the Scala Steward code it seems that it relies on Coursier to retrieve the list of versions available for the artifact.

https://github.com/scala-steward-org/scala-steward/blob/a2cea0f1f72f13b38f3b5cb01ae3cad04a58aff0/modules/core/src/main/scala/org/scalasteward/core/coursier/VersionsCache.scala#L54

If I understand it right, this is done for artifacts that have a maven-metadata.xml file, but as you know, sbt plugins that are published Maven style don’t have one. Which again, to my understanding because of this, Coursier then relies on the directory listing? This is how it seems to work for our maven-central mirror with the sbt-scalafmt plugin. By looking at it with a path like this, I can see an html directory listing of all versions:

https://nexus-redacted.com/repository/maven-public/org/scalameta/sbt-scalafmt_2.12_1.0

Scala Steward (Coursier) is able to correctly gather the versions (again I assume from this and sounds like that from this thread in Steward). However, I’m finding a lot of contradictory information about whether or not this is available on different versions of Nexus3. Plus those that manage our Nexus is saying there is no option to turn this on.

• Community post saying that it exists but is at a different path now <-- This seems to be the case
• Sonatype ticket to create this feature
• Another ticket for the raw tree view
• Announcement saying it’s now supported in 3.7
• SO post saying it’s in a different location now

From what I can tell, for our own internal nexus where I’d expect the directory listing to be like the sbt-scalafmt example above, it just 404’s.

https://nexus-redacted.com/repository/redacted-releases/redacted/redacted/sbt-redacted_2.12_1.0

There is an html index of sorts at this url:

https://nexus-redacted.com/service/rest/repository/browse/redacted-releases/redacted/redacted/sbt-redacted_2.12_1.0/

However, looking through the code here, I don’t believe that will work: https://github.com/coursier/coursier/blob/59bf7739079f6b906ecb77435c8afce2d3658f2c/modules/core/shared/src/main/scala/coursier/maven/MavenRepository.scala#L208-L252

And when Scala Steward attempts to get the available versions of the plugin, it gets none and therefore doesn’t send in prs for our internal sbt plugins. Which sort of leads me to believe that because of the lack of directory listing there, Coursier isn’t getting the versions? Do you know if it’s possible for Coursier to get the versions of an sbt plugin without the directory listing, or does that have to be there? Or could there be another check possible to capture this alternative html view the same way the other is captured?

EDIT: For the time being we just add in a custom release step to create / update the maven-metadata.xml file, but it’d still be great to see if what I mentioned above is the case or not.

[ckipp01]

If you push a maven-metadata.xml file yourself, as you do, there should be a way to have it picked… It should actually be checked if getting the directory listing fails (via the orElse). Maybe this code doesn’t look for maven-metadata.xml at the right URL? It could be fixed then.

Just to clarify, yes, it does work now with us creating the maven-metadata.xml file, but was curious if there was a solution where having that step to manually create one for sbt plugins on self-hosted nexus would no longer be necessary.

[ckipp01]

So this is a pretty hacky solution, but we essentially just included this as a custom release step (if you use sbt-release).

import java.nio.charset.StandardCharsets
import java.nio.file.Paths
import java.text.SimpleDateFormat
import java.util.Base64

import sbtrelease.ReleasePlugin.autoImport.ReleaseStep
import sbt.Project
import sbt.Keys
import sbt.internal.util.ManagedLogger
import Env._

import scala.xml.Elem
import scala.xml.Node
import scala.xml.NodeSeq
import scala.xml.XML

object MetaDataUpdate {

  /** An extra release step that is sort of hack specifically for an sbt plugin.
    * This step will look to see if there is an existing `maven-metadata.xml` file
    * (which sbt plugins don't produce when published maven style) and add the new
    * version to it or create a new one and push it up.
    */
  val step: ReleaseStep = ReleaseStep { st =>
    val logger       = st.log
    val extracted    = Project.extract(st)
    val name         = extracted.get(Keys.name)
    val org          = extracted.get(Keys.organization)
    val version      = extracted.get(Keys.version)
    val target       = extracted.get(Keys.target)
    val metadataName = "maven-metadata.xml"

    val creds = Base64.getEncoder
      .encodeToString(s"${nexusUsername}:${nexusPassword}".getBytes(StandardCharsets.UTF_8))

    val metadataLoc =
      s"${nexusAddress}/${nexusRepositoryPath}${org.replace(".", "/")}/${name}_2.12_1.0/${metadataName}"

    val outputPath = target.getPath() + metadataName

    val existingResponse = requests.get(
      metadataLoc,
      headers = Map(
        "Authorization" -> s"Basic: ${creds}"
      ),
      check = false
    )

    if (existingResponse.is2xx) {
      val rawText          = existingResponse.text()
      val existingMetadata = XML.loadString(rawText)

      val versions = existingMetadata \\ "version"
      val metadata = createMetadata(org, name, version, Some(versions))

      logger.info("Existing maven-metadata.xml found and being updated...")
      saveAndSend(metadataLoc, metadata, outputPath, creds, logger)
    } else if (existingResponse.statusCode == 404) {
      logger.warn("Existing maven-metadata.xml file not found. Attempting to create a new one...")

      val metadata = createMetadata(org, name, version, None)
      saveAndSend(metadataLoc, metadata, outputPath, creds, logger)
    } else {
      logger.error(s"${existingResponse.statusCode.toString()}: ${existingResponse.statusMessage}")
      // If something goes wrong here we just blow up
      sys.error("Something went wrong when requesting maven-metaldata.xml.")
    }

    st
  }

  private def createMetadata(
      org: String,
      name: String,
      newVersion: String,
      existingVersions: Option[NodeSeq]
  ) = {

    val newVersionNode = <version>{newVersion}</version>
    val versions = existingVersions match {
      case Some(existing) => existing ++ newVersionNode
      case None => newVersionNode
    }

    <metadata modelVersion="1.1.0">
      <groupId>{org}</groupId>
      <artifactId>{name}_2.12_1.0</artifactId>
      <versioning>
        <latest>{newVersion}</latest>
        <release>{newVersion}</release>
        <versions>
          {versions}
        </versions>
        <lastUpdated>
          {new SimpleDateFormat("yyyyMMddHHmmss").format(new java.util.Date)}
        </lastUpdated>
      </versioning>
    </metadata>
  }

  private def saveAndSend(
      metadataLoc: String,
      metadata: Elem,
      path: String,
      creds: String,
      logger: ManagedLogger
  ): Unit = {
    XML.save(path, metadata)

    val putReponse = requests.put(
      metadataLoc,
      headers = Map(
        "Authorization" -> s"Basic: ${creds}"
      ),
      data = Paths.get(path)
    )

    if (!putReponse.is2xx) {
      logger.error(s"${putReponse.statusCode}: ${putReponse.statusMessage}")
      sys.error("Unable to update metadata")
    }
  }
}

It’s pretty self-explanatory, but it will just check to see if there is a metadata file, and if not create one. If it does fine one, it just updates it. It’s using requests-scala for the calls.