Random compiler stack overflow

Compiler version

3.1.0

Description

Once or twice a week, dotty crashes with stack overflow when I make a clean compile. Looks like it’s stuck in a loop:

[error] ## Exception when compiling 552 sources to /home/fg/dev/spider6/core/target/scala-3.1.0/classes
[error] java.lang.StackOverflowError
[error] dotty.tools.dotc.transform.Constructors.dotty$tools$dotc$transform$Constructors$$noDirectRefsFrom(Constructors.scala:117)
[error] dotty.tools.dotc.transform.Constructors$intoConstr$1$.transform(Constructors.scala:160)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1396)
[error] dotty.tools.dotc.transform.Constructors$intoConstr$1$.transform(Constructors.scala:160)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1390)
[error] dotty.tools.dotc.transform.Constructors$intoConstr$1$.transform(Constructors.scala:160)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1396)
[error] dotty.tools.dotc.transform.Constructors$intoConstr$1$.transform(Constructors.scala:160)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1390)
[error] dotty.tools.dotc.transform.Constructors$intoConstr$1$.transform(Constructors.scala:160)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1396)
[error] dotty.tools.dotc.transform.Constructors$intoConstr$1$.transform(Constructors.scala:160)

this continues on for 600++ lines.

Interestingly, if I immediately redo the clean / compile cycle inside the same SBT session, without changing anything, it works.

My .sbtopts settings, although seeing the stack trace I don’t think the problem is there:

cat .sbtopts 
-J-Xmx4g
-J-XX:ReservedCodeCacheSize=1g
-J-XX:MaxMetaspaceSize=2g

The full sbt session is bug2.txt

Sorry for the subpar report 🙄️

[fgoret]

Agree 100% on minimisation… This is a difficult bug to catch: 1 - appears at random: only once / twice in a week, then nothing for one or two months, then comes back. 2 - immediately disappears: after the crash, if I say “compile” to sbt, even without “clean”, the stack overflow don’t shows again 3 - no indication from compiler on which line / which file, made it crash. 4 - the whole program is 120,000 lines long

Non objective feeling This stack overflow crash “smells” like improper sharing of memory between multiple threads. Need lots of code to compile, multiple sub-projects (–> concurrent running threads in dotty), to show up only in the unlikely event of two processors running a same piece of code with some mutable shared state.

Now I will make a “dotty stress test” project:

1. Use 5 lines shell script that loops on sbt "; clean; compile" on my customer project, let it run in background until the bug comes back.
2. In the meantime, write a program generator that will create a mock Scala project with thousands of files, a dozen of subprojects, and test it with repeated compilations.
3. Update this issue if / when I will have confirmed results

[som-snytt]

Stack allocation option is -J-Xss2m, that’s the first thing they say on Scala 2. That is, stack to sbt. In case the looping is transiently finite.

There is also a bug fix in that area of code, who knows if it would change this behavior.