Scala Wart: Convoluted de-sugaring of for-comprehensions

Opening this issue, as suggested by Martin, to provide a place to discuss the individual warts brought up in the blog post Warts of the Scala Programming Language and the possibility of mitigating/fixing them in Dotty (and perhaps later in Scala 2.x). These are based on Scala 2.x behavior, which I understand Dotty follows closely, apologies in advance if it has already been fixed

---

Scala lets you write for-comprehensions, which are converted into a chain of flatMaps an maps as shown below:

@ val (x, y, z) = (Some(1), Some(2), Some(3))
x: Some[Int] = Some(1)
y: Some[Int] = Some(2)
z: Some[Int] = Some(3)
@ for{
    i <- x
    j <- y
    k <- z
  } yield i + j + k
res40: Option[Int] = Some(6)
@ desugar{
    for{
      i <- x
      j <- y
      k <- z
    } yield i + j + k
  }
res41: Desugared = x.flatMap{ i => 
  y.flatMap{ j => 
    z.map{ k => 
      i + j + k
    }
  }
}

I have nicely formatted the desugared code for you, but you can try this yourself in the Ammonite Scala REPL to verify that this is what the for-comprehension gets transformed into.

This is a convenient way of implementing nested loops over lists, and happily works with things that aren’t lists: Options (as shown above), Futures, and many other things.

You can also assign local values within the for-comprehension, e.g.

@ for{
    i <- x
    j <- y
    foo = 5
    k <- z
  } yield i + j + k + foo
res42: Option[Int] = Some(11)

The syntax is a bit wonky (you don’t need a val, you can’t define defs or classes or run imperative commands without _ = println("debug")) but for simple local assignments it works. You may expect the above code to be transformed into something like this

res43: Desugared = x.flatMap{ i => 
  y.flatMap{ j =>
    val foo = 5
    z.map{ k => 
      i + j + k
    }
  }
}

But it turns out it instead gets converted into something like this:

@ desugar{
    for{
      i <- x
      j <- y
      foo = 5
      k <- z
    } yield i + j + k + foo
  }
res43: Desugared = x.flatMap(i => 
  y.map{ j =>
    val foo = 5
    scala.Tuple2(j, foo)
  }.flatMap((x$1: (Int, Int)) => 
    (x$1: @scala.unchecked) match {
    case Tuple2(j, foo) => z.map(k => i + j + k + foo)
    }
  )
)

Although it is roughly equivalent, and ends up with the same result in most cases, this output format is tremendously convoluted and wastefully inefficient (e.g. creating and taking-apart unnecessary Tuple2s). As far as I can tell, there is no reason at all not to generated the simpler version of the code shown above.

PostScript:

Notably, these two desugarings do not always produce the same results! The current desugaring behaves weirdly in certain cases; here is one that just bit me an hour ago:

Welcome to Scala 2.11.11 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_112).
Type in expressions for evaluation. Or try :help.

scala>  for{
     |    x <- Right(1).right
     |    y = 2
     |    z <- Right(3).right
     |  } yield x + y + z
<console>:13: error: value flatMap is not a member of Product with Serializable with scala.util.Either[Nothing,(Int, Int)]
          x <- Right(1).right
            ^
<console>:16: error: type mismatch;
 found   : Any
 required: String
        } yield x + y + z
                    ^

This specific problem has gone away in 2.12 because Either doesn’t need .right anymore, but the language-level problem is still there: y = 2 ends up causing strange, difficult-to-debug errors due to the weirdness of the desugaring. This would not be an issue at all given the desugaring I proposed.

[lihaoyi]

I think the correct answer to the confusion is to get people used to the fact that you can’t always filter on a monad. There is a huge number of monads for which filter doesn’t make sense: State, Reader, Writer, Function0, etc… In a way I feel that Scala’s “put .filter on anything” convention is itself a bit of a wart

[LPTK]

Can we at least do something about imperative for comprehensions? They have no justification for generating such horrible code.

The following:

for { x <- xs; y = x; if y > 0; z <- zs } println(x + z)

should generate:

xs.foreach { x =>
  val y = x
  if (y > 0) zs.foreach { z =>
    println(x + z)
  }
}

instead of the current convoluted and inefficient:

xs.map { x =>
  val y = x
  (x, y)
}.withFilter {
  case (x,y) => y > 0
}.foreach {
  case (x,y) => zs.foreach { z => println(x + z) }
}

which creates an intermediate list, allocates tuples, and calls four closure-consuming methods instead of just 2.

No wonder people are surprised how much more efficient it is to use while loops rather than for loops in Scala 😅

I believe the fix for this one is a low-hanging fruit. I could try to do it at some point if people agree with the idea.