Incorrect classpath precedence for scala-xml

While working on fixing a serialization bug in scala-xml, I have been able to convince myself that sbt 0.13.16 incorrectly places the project’s scala-xml version per libraryDependencies AFTER the version of scala-xml that the scala compiler happens to pull in for sbt to use itself.

steps

The high level procedure is you need to publish a local build of scala-xml and pull a sample project you can demo the behavior on.

# Pull, build, publish my branch of scala-xml
git clone https://github.com/scala/scala-xml.git
cd ./scala-xml/
git remote add joescii https://github.com/joescii/scala-xml.git
git fetch joescii
git checkout -b nodeseq-serialization joescii/nodeseq-serialization 
sbt +publishLocal

# Pull and compile my project reproducing the issue
cd ..
git clone https://github.com/joescii/sbt-reproduced.git
cd ./sbt-reproduced/
git fetch origin
git checkout scala-xml
sbt compile

# sbt run will throw an exception because an older scala-xml jar with a serialization bug is used
sbt run

# sbt stage + native launch works perfectly
sbt stage
./target/universal/stage/bin/sbt-reproduced 

problem

The application should run in sbt the same way it runs outside. Using sbt run, you’ll get something like the following:

descartes:sbt-reproduced joescii$ sbt run
[warn] Executing in batch mode.
[warn]   For better performance, hit [ENTER] to switch to interactive mode, or
[warn]   consider launching sbt without any commands, or explicitly passing 'shell'
[info] Loading global plugins from /Users/joescii/.clients/.sbt/0.13/plugins
[info] Loading project definition from /private/tmp/git/sbt-reproduced/project
[info] Set current project to sbt-reproduced (in build file:/private/tmp/git/sbt-reproduced/)
[info] Running code 
Serializing NodeSeq.Empty...
[error] (run-main-0) java.io.NotSerializableException: scala.xml.NodeSeq$$anon$1
java.io.NotSerializableException: scala.xml.NodeSeq$$anon$1

	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1182)
	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
	at code$.serialize$1(code.scala:11)
	at code$.roundTrip(code.scala:22)
	at code$.delayedEndpoint$code$1(code.scala:26)
	at code$delayedInit$body.apply(code.scala:6)
	at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
	at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
	at scala.App$$anonfun$main$1.apply(App.scala:76)
	at scala.App$$anonfun$main$1.apply(App.scala:76)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
	at scala.App$class.main(App.scala:76)
	at code$.main(code.scala:6)
	at code.main(code.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
[trace] Stack trace suppressed: run last compile:run for the full output.
java.lang.RuntimeException: Nonzero exit code: 1
	at scala.sys.package$.error(package.scala:27)
[trace] Stack trace suppressed: run last compile:run for the full output.
[error] (compile:run) Nonzero exit code: 1
[error] Total time: 0 s, completed Aug 4, 2017 11:47:07 AM

expectation

Running it after packaging with sbt-native-packager yields the expected result:

descartes:sbt-reproduced joescii$ ./target/universal/stage/bin/sbt-reproduced 
Serializing NodeSeq.Empty...

See? It works!!

notes

sbt version: 0.13.16 macOS Sierra 10.12.6

Versions of everything:

descartes:sbt-reproduced joescii$ java -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)
descartes:sbt-reproduced joescii$ brew info sbt
sbt: stable 0.13.16, devel 1.0.0-RC2
Build tool for Scala projects
http://www.scala-sbt.org
/usr/local/Cellar/sbt/0.13.15 (378 files, 63.3MB)
  Built from source on 2017-05-22 at 09:19:37
/usr/local/Cellar/sbt/0.13.16 (384 files, 63.5MB) *
  Built from source on 2017-08-04 at 11:46:50

[ashawley]

FWIW, Mark Harrah left us a note:

https://github.com/sbt/sbt/wiki/Scala-modularization-and-classpaths

It seems relevant. There was also this associated discussion on scala-internals

https://groups.google.com/forum/#!msg/scala-internals/tT1pjH5GECE/1wtOZoG9bgQJ

[eed3si9n]

Looking at this again. I think the problem is that during run sbt constructs a classloader that puts all JARs associated with scalaVersion and calls main. That’s how we approximate run.

https://github.com/sbt/zinc/blob/v1.1.1/internal/zinc-classpath/src/main/scala/sbt/internal/inc/classpath/ClasspathUtilities.scala#L66-L72

  def createClasspathResources(classpath: Seq[File], instance: ScalaInstance): Map[String, String] =
    createClasspathResources(classpath, instance.allJars)

  def createClasspathResources(appPaths: Seq[File], bootPaths: Seq[File]): Map[String, String] = {
    def make(name: String, paths: Seq[File]) = name -> Path.makeString(paths)
    Map(make(AppClassPath, appPaths), make(BootClassPath, bootPaths))
  }

This puts compiler JARs into boot classpath. Is boot classpath for taking higher precedence than rt.jar? If so we’d likely just need Scala library here, and not the whole family (compiler, reflect)?