Type promotion rules

This issue seeks to come to a consensus on a subset of type promotion rules (i.e., the rules governing the common result type for two array operands during an arithmetic operation) suitable for specification.

As initially discussed in https://github.com/data-apis/array-api/issues/13, a universal set of type promotion rules can be difficult to standardize due to the needs/constraints of particular runtime environments. However, we should be able to specify a minimal set of type promotion rules which all specification conforming array libraries can, and should, support.

Prior Art

• NumPy: promotion rules follow a type hierarchy (where complex > floating > integral > boolean). See promote_types and result_type APIs and source [1, 2].
• CuPy: follows NumPy’s rules, except for zero-dimension arrays.
• Dask: follows NumPy’s rules.
• JAX: type promotion table (and source) which deviates from NumPy’s promotion rules in two ways: (1) biased toward half- and single-precision floating-point numbers and (2) support for a non-standard floating-point type.
• PyTorch: promotion rules follow a type hierarchy (where complex > floating > integral > boolean) without inspection of value magnitude.
• Tensorflow: requires explicit casting.

Proposal

This issue proposes to specify that all specification conforming array libraries must, at minimum, support the following type promotions:

• floating-point type promotion table:

  	f2	f4	f8
  f2	f2	f4	f8
  f4	f4	f4	f8
  f8	f8	f8	f8

  where

  • f2: half-precision (16-bit) floating-point number
  • f4: single-precision (32-bit) floating-point number
  • f8: double-precision (64-bit) floating-point number

• unsigned integer type promotion table:

  	u1	u2	u4	u8
  u1	u1	u2	u4	u8
  u2	u2	u2	u4	u8
  u4	u4	u4	u4	u8
  u8	u8	u8	u8	u8

  where

  • u1: 8-bit unsigned integer
  • u2: 16-bit unsigned integer
  • u4: 32-bit unsigned integer
  • u8: 64-bit unsigned integer

• signed integer type promotion table:

  	i1	i2	i4	i8
  i1	i1	i2	i4	i8
  i2	i2	i2	i4	i8
  i4	i4	i4	i4	i8
  i8	i8	i8	i8	i8

  where

  • i1: 8-bit signed integer
  • i2: 16-bit signed integer
  • i4: 32-bit signed integer
  • i8: 64-bit signed integer

• mixed unsigned and signed integer type promotion table:

  	u1	u2	u4
  i1	i2	i4	i8
  i2	i2	i4	i8
  i4	i4	i4	i8

Notes

• The minimal set of type promotions outlined above explicitly does not define promotions between types which are not of the same kind (i.e., floating-point versus integer). When converting between types of different kinds, libraries tend to support C type promotion semantics, where floating-point, regardless of precision, has a higher rank/precedence than all integer types; however, they differ in the promoted floating-point precision (e.g., JAX promotes (i8, f2) to f2, while NumPy promotes (i8, f2) to f8). The reason for the discrepancy stems from the particular needs/constraints of accelerator devices, and, thus, by omitting specification here, we allow for implementation flexibility and avoid imposing undue burden.
• Omitted from the above tables are “unsafe” promotions. Notably, not included are promotion rules for mixed signed/unsigned 64-bit integers i8 and u8. NumPy and JAX both promote (i8, u8) to f8 which is explicitly undefined via the aforementioned note regarding conversions between kinds and also raises questions regarding inexact rounding when converting from a 64-bit integer to double-precision floating-point.
• This proposal addresses type promotion among array operands, including zero-dimensional arrays. It remains to be decided whether “scalars” (i.e., non-array operands) should directly participate in type promotion.

[shoyer]

3. Are zero-dimensional arrays considered scalars for the purposes of type promotion?

This is definitely the hardest part of this design decision:

(a) Treating 0d arrays like scalars is convenient because it allows for converting Python scalars into 0d arrays without any impact on promotion rules. You can write the equivalent of np.asarray() on all inputs and not worry about non-array objects again. (b) On the other hand, it introduces an inconsistency between arrays of different ranks. This makes it harder to write (or type check) rank polymorphic code. For example, now add(Array[float32], Array[float64]) -> Array[float64] would fail to type check, because if the float64 input is 0d, it wouldn’t participate in type casting rules and the output would be float32.

My personal opinion is that (b) is a fatal flaw with special casing 0d arrays, and there are good alternatives to immediately calling asarray on inputs (e.g., call np.result_type() on inputs to figure out the result type before casting with np.asarray()).

(Perhaps not coincidentally, JAX answers these questions “No, No, No”)

[rgommers]

This was agreed upon, and the Type Promotion section currently summarizes the 0-D array / scalar issue as:

• Type promotion rules strictly apply when determining the common result type for two array operands during an arithmetic operation, regardless of array dimension. Accordingly, zero-dimensional arrays ar e subject to the same type promotion rules as dimensional arrays.
• Non-array (“scalar”) operands are not permitted to participate in type promotion.

That seems in agreement with the discussion here and complete, so closing this issue.