Checksum error from private Nexus

I’m not really sure what to make of this one. I really don’t know if it’s Coursier downloading the wrong thing, or our Nexus serving the wrong thing, or some combination of the two, but it’s weird.

I have an artifact which is defined by a version range. Specifically: verizon.inf.delorean:core_2.11:1.1.+. There are several versions in this range available, but it isn’t a very actively published project, so nothing new has been pushed in several weeks. I also have several other artifacts which are non-deterministically exhibiting this problem. I just picked this one. Anyway, here’s the error I get:

[error] (core/*:coursierResolution) coursier.ResolutionException: Encountered 1 error(s) in dependency resolution:
[error]   verizon.inf.delorean:core_2.11:1.1.+:
[error]     not found:
[error]       ~/.ivy2/local/verizon.inf.delorean/core_2.11
[error]       https://repo1.maven.org/maven2/verizon/inf/delorean/core_2.11/maven-metadata.xml
[error]       https://oss.sonatype.org/content/repositories/releases/verizon/inf/delorean/core_2.11/maven-metadata.xml
[error]       http://nexus.oncue.verizon.net/nexus/content/groups/internal-ivy/verizon.inf.delorean/core_2.11/
[error]     wrong checksum: ~/.coursier/cache/v1/http/nexus.oncue.verizon.net/nexus/content/groups/internal/verizon/inf/delorean/core_2.11/maven-metadata.xml

If I wipe the ~/.coursier directory and try again, generally it works. Or at least, a different artifact(s) may fail. I can also just delete this specific artifact and it will generally work.

What’s happening is the maven-metadata.xml.sha1 file contains the wrong hash. As far as I can tell, it doesn’t contain the right hash for any of the xml files downloaded by Coursier, nor does it match the hash of any of the xml files in my entire Ivy2 cache. It’s basically just… wrong. The maven-metadata.xml file looks right, and matches the contents of the maven-metadata.xml file from the appropriate artifact path in our Nexus. Additionally, if I curl the maven-metadata.xml.sha1 file which is also in the right place in our Nexus, I get a file which contains the right hash.

So it’s sort of like Coursier just made up a hash that was wrong, stuck it in a file, and complains about it not matching. 😃 Once this situation arises by the way (which is far more often than not with artifacts resolved from this Nexus), it never seems to fix itself. Coursier is just permanently stuck with this artifact never matching its hash.

Again, I don’t know if this is the fault of our Nexus in some way, or if Coursier legitimately has a bug, but it’s pretty annoying. 😃

[eryshev]

So i’ve finished with:

import coursier.CoursierPlugin.autoImport._

coursierChecksums := Nil
coursierArtifactsChecksums := Nil

in my ~/.sbt/0.13/coursier.sbt and it works.

[djspiewak]

Oh I forgot to update the issue…

So I found the root cause of this, and also a workaround. First, the workaround:

coursierChecksums := Nil

Put that in every build.sbt file, or add it to the default settings of every project. Using in ThisBuild doesn’t work for some reason, but that setting does work. Now, for the root cause…

Any Nexus instance which is load-balanced is going to have some metadata syncing between the instances when artifacts are published. This sync has a (sometimes significant!) lag, due to the way that Nexus is designed. So one instance will have a revised artifact-level metadata XML file, while the other will still have the old file. Coursier necessarily makes two separate connections to acquire the metadata XML and its checksum. If these two connections are load balanced to separate instances while the instances are out of sync, Coursier will acquire either stale metadata or a stale checksum. It may actually acquire both in a stale state, but that doesn’t lead to errors.

Of course, Ivy isn’t immune to this either, so I think one of two things are happening. Either Ivy is more forgiving of checksum failure on the artifact-level metadata, or it simply attempts to re-fetch the metadata or its checksum (or both) a few times in the event of a checksum failure. With Coursier, the metadata for that artifact is effectively permanently corrupted (well, until the refresh timeout), since from Coursier’s perspective, the metadata has already been fetched!

@alexarchambault I think one solution that could be explored here is simply immediately expiring the metadata for any artifacts where the checksum doesn’t match. That would still result in a build failure, but at least users could retry. A slightly more sophisticated solution would involve some retries.