[RFC][Relay] Implicit type casting vs explicit dtype conversion

I found two issues in inferring the output dtype of Relay’s type inference.

• For some unary operations such as sqrt, log, exp, the type inference directly uses the input dtype as output dtype. However, if the input type is int type, this inference fails.
• All binary operations now enforeces both operands have the same dtype. However, it is possible that a float tensor adds a int tensor. But it is not allowed in Relay. Though one can argue this is a good language feature for strict type constraint, imported models from existing frameworks could generate such operations.

I suggest two changes to the current type inference in Relay.

• For unary ops like sqrt, log, exp, don’t specify the output dtype. Instead only provide a hint that the output could be a float type. Let the subsequent ops specify whether the output dtype is float32, float64, float16, etc.
• To solve the second issue, there are two options. (I prefer the first option as it is more systemical.)
  • Disable the enforcement that both operands have the same dtype. Implicit cast the dtype for the output dtype. Notice that, implicit cast also exists in the HalideIR.
  • Check the dtype in Relay’s frontend converter, and cast the operand’s dtype whenever they’re different.

[tqchen]

I agree there are tradeoffs that need to be made. This is like the same trade-off we have to make when thinking about broadcasting in binary operators. Let us list the pros and cons

Pros of Implicit Casting

• it makes frontend building relatively easy when dealing with mixed-dtype code

Pros of Explicit Casting

• Mixed dtype is relatively rare in deep learning, so one can argue explicit casting is fine.
• It makes support user-defined new floating type easy(e.g. bfloat), which is crucial for accelerators.
• It simplifies the codegen support in the IR

[icemelon]

Yes, I agree that we should just keep the explicit casting.