Failure in incremental bootstrapped builds

Since a few days (I guess since c9f782cf9d86ed12d83d338b165c3561007a22ec and #4857), I get non-repeatably traces such as, which are only worked around by a clean compilation:

info] No tests to run for dotty-sbt-bridge-bootstrapped / Test / testOnly
[error] ## Exception when compiling 73 sources to /Users/pgiarrusso/git/dotty/compiler/../out/bootstrap/dotty-compiler-bootstrapped/scala-0.10/test-classes
[error] No such file or directory
[error] java.io.UnixFileSystem.createFileExclusively(Native Method)
[error] java.io.File.createTempFile(File.java:2024)
[error] dotty.tools.sbtplugin.TastyFileManager.move(TastyFileManager.scala:59)
[error] dotty.tools.sbtplugin.TastyFileManager.$anonfun$delete$2(TastyFileManager.scala:33)
[error] scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:32)
[error] scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:29)
[error] scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:194)
[error] dotty.tools.sbtplugin.TastyFileManager.delete(TastyFileManager.scala:32)
[error] xsbti.compile.WrappedClassFileManager.delete(WrappedClassFileManager.java:41)
[error] sbt.internal.inc.Incremental$.prune(Incremental.scala:123)
[error] sbt.internal.inc.IncrementalCommon.recompileClasses(IncrementalCommon.scala:107)
[error] sbt.internal.inc.IncrementalCommon.cycle(IncrementalCommon.scala:57)
[error] sbt.internal.inc.Incremental$.$anonfun$compile$3(Incremental.scala:84)
[error] sbt.internal.inc.Incremental$.manageClassfiles(Incremental.scala:129)
[error] sbt.internal.inc.Incremental$.compile(Incremental.scala:75)
[error] sbt.internal.inc.IncrementalCompile$.apply(Compile.scala:61)
[error] sbt.internal.inc.IncrementalCompilerImpl.compileInternal(IncrementalCompilerImpl.scala:309)
[error] sbt.internal.inc.IncrementalCompilerImpl.$anonfun$compileIncrementally$1(IncrementalCompilerImpl.scala:267)
[error] sbt.internal.inc.IncrementalCompilerImpl.handleCompilationError(IncrementalCompilerImpl.scala:158)
[error] sbt.internal.inc.IncrementalCompilerImpl.compileIncrementally(IncrementalCompilerImpl.scala:237)
[error] sbt.internal.inc.IncrementalCompilerImpl.compile(IncrementalCompilerImpl.scala:68)
[error] sbt.Defaults$.compileIncrementalTaskImpl(Defaults.scala:1443)
[error] sbt.Defaults$.$anonfun$compileIncrementalTask$1(Defaults.scala:1417)

Apparently, that’s because a single TastyFileManager can be reused after complete is called: then, we use tempDir after complete has removed it. Logs with instrumentation (after the fix) witnessing this (beware the different instance pointers):

[info] Packaging /Users/pgiarrusso/git/dotty/compiler/target/scala-2.12/dotty-compiler_2.12-0.10.0-bin-SNAPSHOT-nonbootstrapped.jar ...
[info] Done packaging.
ClassFileManager.tempDir: creating /var/folders/kb/bmw9vpm93vl3d5tyc15v8rp40000gn/T/backup5376822862912513440 in dotty.tools.sbtplugin.TastyFileManager@2f7fcfe4
ClassFileManager.delete: maybe using /var/folders/kb/bmw9vpm93vl3d5tyc15v8rp40000gn/T/backup5376822862912513440 in dotty.tools.sbtplugin.TastyFileManager@2f7fcfe4
[info] Compiling 1 Scala source to /Users/pgiarrusso/git/dotty/library/../out/bootstrap/dotty-library-bootstrapped/scala-0.10/classes ...
[info] Done compiling.
ClassFileManager.complete(true): deleting /var/folders/kb/bmw9vpm93vl3d5tyc15v8rp40000gn/T/backup5376822862912513440 from dotty.tools.sbtplugin.TastyFileManager@2f7fcfe4
ClassFileManager.tempDir: creating /var/folders/kb/bmw9vpm93vl3d5tyc15v8rp40000gn/T/backup2539411289080354206 in dotty.tools.sbtplugin.TastyFileManager@cb2bec6
ClassFileManager.delete: maybe using /var/folders/kb/bmw9vpm93vl3d5tyc15v8rp40000gn/T/backup2539411289080354206 in dotty.tools.sbtplugin.TastyFileManager@cb2bec6
[info] Compiling 2 Scala sources to /Users/pgiarrusso/git/dotty/compiler/../out/bootstrap/dotty-compiler-bootstrapped/scala-0.10/classes ...
[info] Done compiling.
ClassFileManager.complete(true): deleting /var/folders/kb/bmw9vpm93vl3d5tyc15v8rp40000gn/T/backup2539411289080354206 from dotty.tools.sbtplugin.TastyFileManager@cb2bec6
ClassFileManager.tempDir: creating /var/folders/kb/bmw9vpm93vl3d5tyc15v8rp40000gn/T/backup7561286245069696276 in dotty.tools.sbtplugin.TastyFileManager@cb2bec6
ClassFileManager.delete: maybe using /var/folders/kb/bmw9vpm93vl3d5tyc15v8rp40000gn/T/backup7561286245069696276 in dotty.tools.sbtplugin.TastyFileManager@cb2bec6
[info] Compiling 1 Scala source to /Users/pgiarrusso/git/dotty/compiler/../out/bootstrap/dotty-compiler-bootstrapped/scala-0.10/test-classes ...
[info] Done compiling.
ClassFileManager.complete(true): deleting /var/folders/kb/bmw9vpm93vl3d5tyc15v8rp40000gn/T/backup7561286245069696276 from dotty.tools.sbtplugin.TastyFileManager@cb2bec6

IIUC, @jvican suggested that the lifecycle of external ClassFileManager (which TastyFileManager is) might be different from the one of internal ones. That would explain why this bug appears here but not in TransactionalClassFileManager (which uses the same pattern).

[allanrenucci]

Note to myself: generatedTastyFiles and movedTastyFiles should be cleared on complete https://github.com/lampepfl/dotty/blob/e8f137cab384bfa96d94d2817a2962bb9282298b/sbt-dotty/src/dotty/tools/sbtplugin/TastyFileManager.scala#L31-L32

[Blaisorblade]

@allanrenucci Yes, that fits with my understanding and @jvican’s comments on Gitter.

(if that’s your question the answer is yes, the external class file manager is reused because ExternalHooks requires an instance of it)