Reducing open types is potentially unsafe

The following currently causes a stack overflow in dotty (it’s rejected by scalac):

trait A { type L[X] }
trait B { type L }
trait C { type M <: A }
trait D { type M >: B }

object Test {
  def test(x: C with D): Unit = {
    def f(y: x.M)(z: y.L[y.L]) = z
    f(new B { type L[F[_]] = F[F] })(1)
  }
}

The overflow is caused by the compiler attempting to evaluate the non-terminating type expression y.L[y.L] which is ill-kinded.

The type definition type L[F[_]] = F[F] } is clearly ill-kinded already and scalac actually reports this.

But the deeper issue is in the line above that definition: the type y.L has two conflicting kinds, it is both a type operator and a proper type. This is because L is declared as a type operator in trait A and as a proper type in trait B, which are the upper and lower bounds of x.M, respectively. So we can construct an instance y of B where y.L is a proper type and then apply it to some other type by up-casting y to x.M. Here’s a variant of the example which illustrates that point more clearly:

//...
object Test {
  def test(x: C with D): Unit = {
    def f(y: x.M)(z: y.L[Any]) = z  // <-- y.L used as a type operator
    f(new B { type L = Any })(1)    // <-- y.L defined as a proper type
  }
}

It is rejected with the rather absurd error message

-- [E007] Type Mismatch Error:
 9 |		f(new B { type L = Any })(1)
   |		                          ^
   |		                          found:    Int(1)
   |		                          required: Any[Any]

Changing the type of z to y.L results instead in

-- [E055] Syntax Error:
 8 |    def f(y: x.M)(z: y.L) = z
   |                     ^^^
   |                     missing type parameter for y.L

This is another instance of a know problem about type members with absurd bounds (see also issue #50): evaluation is unsafe in a context containing variables/values (here x) of a type (here C & D) containing type members with absurd bounds (here x.M). In this particular case, the unsafe evaluation happens in an open type (namely y.L[Any]) rather than an open term, and at compile time (rather than at runtime).

Clarification: by “open type” I mean a type that contains free type or term variables (as opposed to a closed type which contains no free variables).

[sstucki]

Good points @Blaisorblade.

As you say, reduction of open types is not (kind-)safe and can get stuck/fail. Yet, compiling some examples needs reduction of open types (see below). So it seems some of the failures arising from bad types must be detected before leading to such bad behavior, doesn’t it?

Yes, this seems like a problem and I don’t know how best to address it.

Regarding “undefined kinds” — I expect some code needs to know the (most specific) kind of y.L in your example, but there is no such thing. Hence “undefined kind”. Of course, I have to be informal here, so I’ll need some slack.

OK, I’ll use that slack to interpret your “most specific kind” as “synthesized kind” and then it seems the only viable option here is * -> *. It should of course depend on the synthesized type of y which is itself a type selection, namely x.M. So to allow a type selection on y the type checker has to do some widening and I’m really not familiar enough with the compiler internals to tell you how this is done. My guess, though, is that x.M will be widened to A (rather than Any) in which case y.L is considered a type operator. But I might be wrong. Maybe @smarter or @odersky could shed some light on this?

Even if you prove String <: Int, will Dotty try treating a String as an Int? It seems not yet, but you can try.

At least as of 6f8bc43, dotc -optimise doesn’t break yet on the following code, but if it’s not in it should probably be added as a testcase—maybe you can ping whoever maintains the simplifier and they can do better?

That is a very good point. Evaluation of open terms is of course not safe either, which should be a problem for optimizations such as constant folding, as you suggest. I haven’t tried this but it could come down to a final modifier on typesafeCast or something of that sort. Maybe @DarkDimius has some insights here?

[sstucki]

This is likely related, though the error is different: #2887.