Incremental runs lead to GCC reporting a JSC_UNDEFINED_VARIABLE

Initially reported in mdoc at https://github.com/scalameta/mdoc/issues/454. mdoc uses the linker incrementally as a matter of course:

• For each .md file, it gathers all the snippets in a single (virtual) .scala source file, with each one in a separate @JSExportTopLevel def inside one object.
• It links the result for each file, and keeps the linker hot between .md files.

Clearly, this exercises the incremental linker in a fairly reproducible way, which led to the above issue to be deterministic.

I managed to further minimize the above report so that it does not depend on mdoc at all, reproducing it in our hello world project.

First, in the build, comment out scalaJSUseMainModuleInitializer := true in the helloworld project. Then, open sbt.

Paste the following code in HelloWorld.scala, save, and run helloworld2_12/fullOptJS:

object mdocjs {
  @_root_.scala.scalajs.js.annotation.JSExportTopLevel("mdoc_js_run1")
  def mdoc_js_run1(): Unit = {
    import scala.util.Success
    Success("foo")
  }

  @_root_.scala.scalajs.js.annotation.JSExportTopLevel("mdoc_js_run2")
  def mdoc_js_run2(): Unit = {
    import scala.util.Success
    Success("bar")
  }
}

Repeat with the following code (do not exit or reload sbt in-between):

object mdocjs {
  @_root_.scala.scalajs.js.annotation.JSExportTopLevel("mdoc_js_run1")
  def mdoc_js_run1(): Unit = {
    println("b")
  }
}

Repeat once more with the following code:

object mdocjs {
  @_root_.scala.scalajs.js.annotation.JSExportTopLevel("mdoc_js_run1")
  def mdoc_js_run1(): Unit = {
    println("c1")
  }

  @_root_.scala.scalajs.js.annotation.JSExportTopLevel("mdoc_js_run2")
  def mdoc_js_run2(): Unit = {
    println("c2")
  }
}

The last invocation triggers the error:

sbt:Scala.js> helloworld2_12/fullOptJS
[info] Compiling 1 Scala source to C:\Users\sjrdo\Documents\Projets\scalajs\examples\helloworld\.2.12\target\scala-2.12\classes ...
[info] Done compiling.
[info] Full optimizing C:\Users\sjrdo\Documents\Projets\scalajs\examples\helloworld\.2.12\target\scala-2.12\helloworld-opt
[error] JSC_UNDEFINED_VARIABLE. variable $c_s_util_Success is undeclared at file:///C:/Users/sjrdo/Documents/Projets/scalajs/examples/helloworld/src/main/scala/helloworld/HelloWorld.scala line 11 : 11
[error] Closure: 1 error(s), 0 warning(s)
[error] There were errors when applying the Google Closure Compiler
[error] (helloworld2_12 / Compile / fullLinkJS) There were errors when applying the Google Closure Compiler

I have not tried to minimize this further than the above yet. One thing to try would be to get rid of the dependency on println and Success, so that we would only need the minilib to reproduce it in the linker tests.

[sjrd]

After a full morning minimizing and producing the above linker test, and a full afternoon debugging, I have finally found the issue and I have a fix. However, this session highlighted a weakness in our Versions, which is worth discussing.

Without further ado, the problem happens when:

• a Public method exists in V1, is removed in V2 (but its enclosing class still exists) and reappears in V3.

it happens because:

• the IncOptimizer uses a counter for the version:
  • created at https://github.com/scala-js/scala-js/blob/92c64df83377d47c55d3de0ff78d10da3ba06bcc/linker/shared/src/main/scala/org/scalajs/linker/frontend/optimizer/IncOptimizer.scala#L831, and incremented before each use at https://github.com/scala-js/scala-js/blob/92c64df83377d47c55d3de0ff78d10da3ba06bcc/linker/shared/src/main/scala/org/scalajs/linker/frontend/optimizer/IncOptimizer.scala#L967-L969
  • it is attached to a particular MethodImpl; when the MethodImpl survives two consecutive runs, the counter is correctly incremented. But if the method disappears in a run, the MethodImpl is discarded, and when it reappears, the counter is reset to 0!
  • this means that in the 1st and 3rd run, the version of run2 is "1", both times
• the Refiner actually correctly deletes its own caches when they are not used in a run (so the method cache is removed in the 2nd run), and so it correctly recreates an updated version of the method in the 3rd run, with the same version (of course) but with the new body
• the Emitter, however, is missing a startRun() for non-static-like methods (basically Public methods) at https://github.com/scala-js/scala-js/blob/92c64df83377d47c55d3de0ff78d10da3ba06bcc/linker/shared/src/main/scala/org/scalajs/linker/backend/emitter/Emitter.scala#L598-L602; this means that it never resets its _cacheUsed flag, and hence its clearAfterRun() never deletes a method
• Hence, in the 2nd run, the cached version of run2 survives. And when we get to the 3rd run, we have a new one, but it has the same version as from the 1st run, and so the Emitter thinks it doesn’t need to regenerate it.

Of course, we should fix the missing startRun() in the Emitter, if only because it means it retains unnecessary stuff in memory. But this analysis raises a larger question: is it OK to reuse a version string for an entity that happened to exist in the past, but disappeared in a run before reappearing in a later run? Clearly, allowing that kind of reuse is sensitive to later caches being always removed when not used in a run, which should not be required for correctness.

@gzm0 Thoughts?

[sjrd]

There is one easy way to generate versions, but it might be slow: actually re-hash the optimized MethodDefs. This was not even an option before, but now Hashers work on JS too.

Another possibility: use a tree of SplittableRandoms that follows the class hierarchy and then each method within a class.