Generalizing Fuse MatMul and Add into GEMM pass

@vloncar @houseroad

I recently was looking at a graph at work and saw a lot of MatMul + Add operations. Here’s a sample image of what I’m talking about:

I tried running the new pass @vloncar added but it didn’t work. The shape output of MatMul is float32[32, 100] and the Initializer in Add is float32[100].

Is the pass not applying because of this portion of the code?

    if (orig_bias->node()->kind() != kConstant &&
        orig_bias->node()->kind() != kParam) {
      return false;
    }

If so why are we doing this?

[ArmenAg]

@houseroad

This is a good segway into having canonicalization in our optimization framework. Before we run any pass we should canonicalize the graph.

I can write a more detailed proposal later, but some basic things to do would be to infer shapes of the model using shape_inference::InferShapes so we don’t have to rely on the frontend exporters, exporting shape information (most don’t by the way). Some other things that we can do is bubble up cast operators through ShapeTransforms (as defines by OpAnnotationFlag) and that way we could have a more advanced cast fusion optimization (btw which is something I’ll add later).

[jcwchen]

Please note that ONNX optimizer has been moved to another repo https://github.com/onnx/optimizer since ONNX 1.9. If you still have this question, please raise an issue there. Thank you!