Incremental compilation is completely broken in 1.3 on Linux

To reproduce this, I’ve created the simplest possible project and set build.properties to the latest sbt nightly: https://github.com/smarter/sbt-inc-broken

% sbt

[info] Loading project definition from /home/smarter/opt/sbt-inc-broken/project
[info] Loading settings for project root from build.sbt ...
[info] Set current project to root (in build file:/home/smarter/opt/sbt-inc-broken/)
[info] sbt server started at local:///home/smarter/.sbt/1.0/server/c00128c57db581cfabf9/sock
[root@master]> compile
[info] Compiling 1 Scala source to /home/smarter/opt/sbt-inc-broken/target/scala-2.12/classes ...
[info] Done compiling.
[success] Total time: 1 s, completed 12 févr. 2019 23:13:54
[root@master]> compile
[info] Compiling 1 Scala source to /home/smarter/opt/sbt-inc-broken/target/scala-2.12/classes ...
[info] Done compiling.
[success] Total time: 0 s, completed 12 févr. 2019 23:13:56
[root@master]> compile
[info] Compiling 1 Scala source to /home/smarter/opt/sbt-inc-broken/target/scala-2.12/classes ...
[info] Done compiling.
[success] Total time: 0 s, completed 12 févr. 2019 23:13:57

Turning on debug logs doesn’t help much:

[root@master]> debug
[root@master]> set incOptions in Global ~= { _.withApiDebug(true).withRelationsDebug(true) }
[root@master]> compile
[debug] Updating ...
[... A ton of useless overly verbose update messages ...]
[debug] Done updating 
[debug] 
[debug] Initial source changes: 
[debug]         removed:Set()
[debug]         added: Set()
[debug]         modified: Set()
[debug] Invalidated products: Set(/home/smarter/opt/sbt-inc-broken/target/scala-2.12/classes/A.class)
[debug] External API changes: API Changes: Set()
[debug] Modified binary dependencies: Set()
[debug] Initial directly invalidated classes: Set()
[debug] 
[debug] Sources indirectly invalidated by:
[debug]         product: Set(/home/smarter/opt/sbt-inc-broken/A.scala)
[debug]         binary dep: Set()
[debug]         external source: Set()
[debug] All sources are invalidated.
[debug] [inv] Invalidate package objects by inheritance only...
[debug] Initial set of included nodes: 
[debug] [inv] Package object invalidations: 
[debug] Recompiling all sources: number of invalidated sources > 50.0% of all sources
[debug] [inv] ********* Pruned: 
[debug] [inv] Relations (with name hashing enabled):
[debug] [inv]   products: Relation [ ]
[debug] [inv]   library deps: Relation [ ]
[debug] [inv]   library class names: Relation [
[debug] [inv]     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar -> java.lang.Object
[debug] [inv] ]
[debug] [inv]   internalDependencies: 
[debug] [inv]     
[debug] [inv]   externalDependencies: 
[debug] [inv]     
[debug] [inv]   class names: Relation [ ]
[debug] [inv]   used names: Relation [ ]
[debug] [inv]   product class names: Relation [ ]
[debug] [inv] *********
[info] Compiling 1 Scala source to /home/smarter/opt/sbt-inc-broken/target/scala-2.12/classes ...
[debug] Getting org.scala-sbt:compiler-bridge_2.12:1.2.1-bin-20190204T212322:compile for Scala 2.12.8
[debug] Getting org.scala-sbt:compiler-bridge_2.12:1.2.1-bin-20190204T212322:compile for Scala 2.12.8
[debug] [zinc] Running cached compiler 3d250bda for Scala compiler version 2.12.8
[debug] [zinc] The Scala compiler is invoked with:
[debug]         -bootclasspath
[debug]         /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/resources.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jsse.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jce.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/charsets.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jfr.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/classes:/home/smarter/.sbt/boot/scala-2.12.8/lib/scala-library.jar
[debug]         -classpath
[debug]         /home/smarter/opt/sbt-inc-broken/target/scala-2.12/classes
[debug] Scala compilation took 0.122393405 s
[info] Done compiling.
[debug] [inv] ********* Fresh: 
[debug] [inv] Relations (with name hashing enabled):
[debug] [inv]   products: Relation [
[debug] [inv]     A.scala -> target/scala-2.12/classes/A.class
[debug] [inv] ]
[debug] [inv]   library deps: Relation [
[debug] [inv]     A.scala -> /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar
[debug] [inv] ]
[debug] [inv]   library class names: Relation [
[debug] [inv]     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar -> java.lang.Object
[debug] [inv] ]
[debug] [inv]   internalDependencies: 
[debug] [inv]     
[debug] [inv]   externalDependencies: 
[debug] [inv]     
[debug] [inv]   class names: Relation [
[debug] [inv]     A.scala -> A
[debug] [inv] ]
[debug] [inv]   used names: Relation [
[debug] [inv]     A -> UsedName(A,[Default])
[debug] [inv]     A -> UsedName(AnyRef,[Default])
[debug] [inv]     A -> UsedName(Object,[Default])
[debug] [inv]     A -> UsedName(java;lang;Object;init;,[Default])
[debug] [inv]     A -> UsedName(scala,[Default])
[debug] [inv] ]
[debug] [inv]   product class names: Relation [
[debug] [inv]     A -> A
[debug] [inv] ]
[debug] [inv] *********
[debug] [inv] ********* Merged: 
[debug] [inv] Relations (with name hashing enabled):
[debug] [inv]   products: Relation [
[debug] [inv]     A.scala -> target/scala-2.12/classes/A.class
[debug] [inv] ]
[debug] [inv]   library deps: Relation [
[debug] [inv]     A.scala -> /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar
[debug] [inv] ]
[debug] [inv]   library class names: Relation [
[debug] [inv]     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar -> java.lang.Object
[debug] [inv] ]
[debug] [inv]   internalDependencies: 
[debug] [inv]     
[debug] [inv]   externalDependencies: 
[debug] [inv]     
[debug] [inv]   class names: Relation [
[debug] [inv]     A.scala -> A
[debug] [inv] ]
[debug] [inv]   used names: Relation [
[debug] [inv]     A -> UsedName(A,[Default])
[debug] [inv]     A -> UsedName(AnyRef,[Default])
[debug] [inv]     A -> UsedName(Object,[Default])
[debug] [inv]     A -> UsedName(java;lang;Object;init;,[Default])
[debug] [inv]     A -> UsedName(scala,[Default])
[debug] [inv] ]
[debug] [inv]   product class names: Relation [
[debug] [inv]     A -> A
[debug] [inv] ]
[debug] [inv] *********

However, if I restart sbt, then compile succeeds without doing anything, right until I do clean once, then it’s back to compile compiling everything all the time.

I’ve tried manually bisecting this by using different sbt nightly builds, but downloading the jars for a new nightly takes several minutes (!) and the couple I tried were broken (are nightly builds published even if the CI fails, or is the CI missing stuff to check that nightly builds are usable ?), so I give up.

/cc @jvican who might know what changed in zinc in 1.3 that could have caused this.

[fooblahblah]

I was going to follow-up when I had completed this CI task and it was actually working, but I’m having success tar’ing between build steps using a little posix flag which keep timestamps down to the nanosecond (I think).

tar -H posix -czvf ../project.tgz .

I was trying to only archive target to save a little time, but from some tinkering I’ve found best results where I use the above command after the build phase. Then downstream I un-tar the whole tree and SBT seems to chug along with the test without compiling.

The command for the tests is:

./sbt -J-Xmx14g "set offline := true" "set skip in update := true" test

I’m still working out a few details, but it’s looking promising. This is with sbt-1.2.8.

Just for background I’m trying to use CircleCI workflows so I can parallel-ize our webpack and scala builds, then run the tests, then deploy to Elastic Beanstalk and ECR in parallel too. Without the archiving I’d get hit by a recompile each time.

Bloop looks promising, but we’re currently dependent on a couple SBT plugins I’d need to extract into scripts before bloop will be usable for us.

[eatkins]

The bug was probably introduced in this commit: https://github.com/sbt/sbt/commit/25e97f99f588d1ddc9bbd114740bfb25c2a955c6 The idea is that we use a global cache of file stamps so that zinc doesn’t have to do io to figure out which files it needs to recompile. It’s possible that zinc is writing stamps to the analysis file in a different format from how sbt is caching them. I should add debug logging of the stamp comparisons to debug issues like this.