Missing value support brought perf regression

I’m working on a blog post for HB + TVM, and I found that the PR https://github.com/microsoft/hummingbird/pull/419/ brought big perf regression compared to the benchmark results I had two months ago.

For example, here is the GPU result on year dataset, before and after https://github.com/microsoft/hummingbird/pull/419/. I’m seeing similar regression on other dataset as well, and pytorch results are also worse.

Before

{'conversion_time': '26.918972336000024', 'prediction_time': '0.005081184775504806', 'peak_mem': '1863.08984375', 'is_same_output': True}
{
  "year": {
    "xgb": {
      "hb-tvm": {
        "conversion_time": "26.918972336000024",
        "is_same_output": true,
        "peak_mem": "1863.08984375",
        "prediction_time": "0.005081184775504806"
      },
      "peak_mem": 856.05078125,
      "prediction_time": 0.017587377367330452
    }
  }
}

After

{'conversion_time': '78.68978722800011', 'prediction_time': '0.006477392132670242', 'peak_mem': '1857.5703125', 'is_same_output': True}
{
  "year": {
    "xgb": {
      "hb-tvm": {
        "conversion_time": "78.68978722800011",
        "is_same_output": true,
        "peak_mem": "1857.5703125",
        "prediction_time": "0.006477392132670242"
      },
      "peak_mem": 861.5234375,
      "prediction_time": 0.017615050061188096
    }
  }
}

I don’t know what https://github.com/microsoft/hummingbird/pull/419/ did, but the graph is definitely bigger than before. Conversion time also increased by more than 2x, but that’s TVM mystery that I forget for now.

Since dataset doesn’t have any missing data, https://github.com/microsoft/hummingbird/pull/419 shouldn’t change anything, and this regression is not acceptable. Can we revert https://github.com/microsoft/hummingbird/pull/419 until regression is fixed? Also, can we setup benchmark infra on CI so that things like wouldn’t happen in the future?

@interesaaat @scnakandala

[masahi]

@ksaur yes it fixed it, thanks for quick action! @interesaaat

[masahi]

Yes, I bisected it and the Before number above is from https://github.com/microsoft/hummingbird/commit/94b584b87bd4ae4374f1c9828a418df201eca62f

Ideally we want a design where missing value support is factored into a single class, and turning it on or off can be done by swapping in or out that class.