Multiversal Equality

Note

The status of eqAny and other predefined eq comparisons has changed in #2786, together with the resolution algorithm. The up-to-date details are in the reference http://dotty.epfl.ch/docs/reference/multiversal-equality.html

Motivation

Scala prides itself of its strong static type system. Its type discipline is particularly useful when it comes to refactoring. Indeed, it’s possible to write programs in such a way that refactoring problems show up with very high probability as type errors. This is essential for being able to refactor with the confidence that nothing will break. And the ability to do such refactorings is in turn very important for keeping code bases from rotting.

Of course, getting such a robust code base requires the cooperation of the developers. They should avoid type Any, casts, stringly typed logic, and more generally any operation over loose types that do not capture the important properties of a value. Unfortunately, there is one area in Scala where such loose types are very hard to avoid: That’s equality. Comparisons with == and != are universal. They compare any two values, no matter what their types are. This causes real problems for writing code and more problems for refactoring it.

For instance, one might want to introduce a proxy for some data structure so that instead of accessing the data structure directly one goes through the proxy. The proxy and the underlying data would have different types. Normally this should be an easy refactoring. If one passes by accident a proxy for the underlying type or vice versa the type checker will flag the error. However, if one accidentally compares a proxy with the underlying type using == or a pattern match, the program is still valid, but will just always say false. This is a real worry in practice. I recently abandoned a desirable extensive refactoring because I feared that it would be too hard to track down such errors.

Current Status

The problems of universal equality in Scala are of course well known. Some libraries have tried to fix it by adding another equality operator with more restricted typing. Most often this safer equality is written ===. While === is certainly useful, I am not a fan of adding another equality operator to the language and core libraries. It would be much better if we could fix == instead. This would be both simpler and would catch all potential equality problems including those related to pattern matching.

How can == be fixed? It looks much harder to do this than adding an alternate equality operator. First, we have to keep backwards compatibility. The ability to compare everything to everything is by now baked into lots of code and libraries. For instance, we might have a Map with Any keys that uses universal equality and hashcode to store and retrieve any value. Second, with just one equality operator we need to make this operator work in all cases where it makes sense. An alternative === operator can choose to refuse some comparisons which would still be sensical because there’s always == to fall back to. With a unique == operator we do not have this luxury.

The current status in Scala is that the compiler will give warnings for some comparisons that are always false. But the coverage is very weak. For instance this will give a warning:

scala> 1 == "abc"
<console>:12: warning: comparing values of types Int and String using `==' will always yield false

But this will not:

scala> "abc" == 1
res2: Boolean = false

There are also cases where a warning is given for a valid equality test that actually makes sense because the result could be true. In summary, the current checking catches some obvious bugs, which is nice. But it is far too weak and fickle to be an effective refactoring aid.

Proposal

I believe to do better, we need to enlist the cooperation of developers. Ultimately it’s the developer who provides implementations of equality methods and who is therefore best placed to characterize which equalities make sense. Sometimes this characterization can be involved. For instance, an Int can be compared to other primitive numeric values or to instances of type Number but any other comparison will always yield false. Or, it makes sense to compare two List values if and only if it makes sense to compare the list’s element values.

The best known way to characterize such relationships is with type classes (aka implicit values). A type class Eq[T, U] can capture the property that values of type T can be compared to values of type U. Here’s the proposed definition of this type class:

package scala

import annotation.implicitNotFound

/** A marker trait indicating that values of kind `T` can be compared to values of type `U`. */
@implicitNotFound("Values of types ${L} and ${R} cannot be compared with == or !=")
sealed trait Eq[-L, -R]

/** Besides being a companion object, this object
 *  can also be used as a value that's compatible with
 *  any instance of `Eq`.
 */
object Eq extends Eq[Any, Any] {

  /** A fall-back implicit to compare values of any types.
   *  The compiler will restrict implicit instances of `eqAny`. An instance
   *  `eqAny[T, U]` is _invalid_ if `T` or `U` is a non-bottom type that
   *  has an implicit `Eq[T, T]` (respectively `Eq[U, U]`) instance.
   *  An implicit search will fail instead of returning an invalid `eqAny` instance,
   */
  implicit def eqAny[L, R]: Eq[L, R] = Eq
}

Note 1: Eq is contravariant. So an instance of, say, Eq[Number, Number] is also an instance of Eq[BigInt, BigDecimal]. In other words: If all instances of Number can be compared then BigInts can be compared to BigDecimals, because they are both instances of Number.

Note 2: Eq does not have any members; it’s a pure marker trait. The idea is that the Scala compiler will check every time it encounters a potentially problematic comparison between values of types T and U that there is an implicit instance of Eq[T, U]. A comparison is potentially problematic if it is between incompatible types. As long as T <: U or U <: T the equality could make sense because both sides can potentially be the same value.

The Scala compiler will also perform this check on every pattern match against a value (i.e. where the pattern is a non-variable identifier or a selection, or a literal). It treats such a pattern match like a comparison between the type of the scrutinee and the type of the pattern value.

Developers can define equality classes by giving implicit Eq instances. Here is a simple one:

   implicit def eqString: Eq[String, String] = Eq

Since Eq does not have any members, the sole purpose of these implicit instances is for type checking. The right hand side of the instance does not matter, as long as it is type-correct. The companion object Eq is always correct since it extends Eq[Any, Any].

Parameterized types generally need polymorphic Eq instances. For example:

  implicit def eqList[T, U](implicit _eq: Eq[T, U]): Eq[List[T], List[U]] = Eq

This expresses that Lists can be compared if their elements can be compared.

What about types for which no Eq instance exists? To maintain backwards compatibility, we allow comparisons of such types as well, by means of the standard eqAny instance in the Eq object that was shown above. The type signature of eqAny suggests that it works for any two types, but this would render equality checking ineffective, because Eq instances would always be found. But in fact the compiler will check any generated eqAny instance in implicit search for validity.

Let lift be a function on types that maps every covariant occurrence of an abstract type to its upper bound and that drops all refinements in covariant positions. An instance

  eqAny[T, U]

is valid if T <: lift(U) or if U <: lift(T) or if both T, U are Eq-free. A type S is Eq-free if there is no implicit instance of type Eq[S, S] other than eqAny itself. Invalid eqAny instances will not returned from implicit search.

Note 3: The purpose of lift above is to make code like this compile:

    def f[T](x: T) = 
      if (x == null) ... 
      else if (x == "abc") ...
      else ...

Without it, we would have to resort to a widening cast, i.e. if (x.asInstanceOf[Any] == null, which is ugly.

Note 4: It is conceivable that the eqAny behavior can be implemented as a macro. We will know more once macros are redesigned.

A Refinement

The scala.equalityClass annotation can be used to avoid the boilerplate of writing implicit Eq instances by hand. If @equalityClass is given for a class C, the companion object of C would get an automatically generated Eq instance defined in the way sketched above. For instance:

  @equalityClass trait Option[+T] { ... }

would generate the following Eq instance:

object Option {
  implicit def eqOption[T, U](implicit $x0: Eq[T, U]): Eq[Option[T], Option[U]] = Eq
}

It should be possible to get this functionality by making @equalityClass a macro annotation.

Properties

Here are some properties of the proposal

1. It is opt-in. To get safe checking, developers have to annotate classes that should allow comparisons only between their instances with @equalityClass, or they have to define implicit Eq instances by hand.
2. It is backwards compatible. Without @equalityClass annotations equality works as before.
3. It carries no run-time cost compared to universal equality. Indeed the run-time behavior of equality is not affected at all.
4. It has no problems with parametricity, variance, or bottom types.
5. Depending on the actual Eq instances given, it can be very precise. That is, no comparisons that might yield true need to be rejected, and most comparisons that will always yield false are in fact rejected.

The scheme effectively leads to a partition of the former universe of types into sets of types. Values with types in the same partition can be compared among themselves but values with types in different partitions cannot.

An @equalityClass annotation on a type creates a new partition. All types that do not have any Eq instances (except eqAny, that is) form together another partition.

So instead of a single universe of values that can be compared to each other we get a multiverse of partitions. Hence the name of the proposal: Multiversal Equality.

Implementation Status

The core of the proposal is implemented in #1246. The refinement of having @equalityClass annotations generate Eq instances is not yet implemented.

Experience

As a first step, I added Eq instances to some of the core types of dotty: Type, Symbol, Denotation, Name. Two bugs were found where comparisons made no sense because they compared values of different types and therefore would always yield false (one of them was a pattern match). There were no instances where a sensical comparison was flagged as an error.

A Possible Variant

The current proposal needs a change to the handling of contravariant type parameters in implicit search and overloading resolution, which is described and implemented in 340f88353bf9a307971fd2d96d5fa8d0a2431bcf. Without that change, eqAny would always take precedence over any other implicit Eq instance, which is not what we want.

If we do not want to rely on that change, one could alternatively make Eq non-variant, but then Eq instances become more complicated because they have to talk about all possible subtypes of an equality class using additional type parameters.

[smarter]

I like the first choice more, it means that I can guarantee that equality works the way I want in my code without worrying about what libraries do.

[Sciss]

@Blaisorblade

Existing code should keep working without changes.

That’s exactly what I’m proposing with import scala.language.universalEquality. That’s exactly the same as saying we need to import now scala.language.implicitConversion if we want to have implicit conversions (one would have to discuss warning vs. error). Existing code with zero changes doesn’t make sense. It’s simple enough to add, as you propose for the opposite flag, that flag to your sbt file. Secondly, we are talking about Dotty not Scala 2.x.

What you are saying is flip things back to Martin’s proposal. So what’s the addition here?

---

So if your old code read

def foo(x: Option[String], b: String) = x == b

That constituted a bug, and thus “breaking” compilation in the new version should be the desired behaviour. So we assume everything in std lib will have @equalityClass, or—as I suggest—nothing will have nothing because this is the default. I can’t see a case where you would have @universalEquality in std.lib.

Then in client code if you have the rare case of

trait Not
trait Related { def equals(that: Any): Boolean = that.isInstanceOf[Not] }

You can still compile it if you add legacy flag -language:universalEquality.

Win-win IMHO.