[Sparse] Support sparse matrix computation

Objective

• To reduce unnecessary work load in computing convolution over large amount of zeros.
• To reduce memory/cache footprint

TODOs

• Step 0 : Summarize the design decision
• Step 1 : Update the PR (#1289) to support the following features
• Create tvm.contrib.sparse.CSRNDArray and tvm.contrib.sparse.placeholder for creating sparse tensor.
• Provide conversion between numpy.ndarray and tvm.contrib.sparse.CSRNDArray
• Implement topi.sparse.csrmv and topi.sparse.csrmm as SpMV and SpMM, and check correctness with dense tensor operations
• Support some other sparse tensor computation (e.g. relu, batch_norm, flatten)
• Demonstrate sparse approximation based on MobileNetV2, and compare performance with dense matrix computation baseline.
• Write a blogpost introducing this feature.

Proposed API Changes

• There will no change to original Tensor object.

[liangfu]

Hi @fredrikbk , thanks for your explanation. I think we should refer to taco for the code generation part in tvm, especially for the unified sparse tensor structure. However, we would very likely to support the runtime for cuda, opencl and many other backends, therefore I think it is necessary to implement code generation in tvm.

In my observation, there are two categories of algorithms that extensively use sparse tensor operations to speed up convolution. First, people depends on high sparsity of weight tensor to speed up 2D image convolution, which requires retraining the network weights to increase the sparsity. Second, sparse tensors are used to represent point cloud dataset and its processing steps, where sparse tensors are used to represent features instead of weights in a convolution operator. In many cases, we might need to implement direct sparse convolution as well.

[tqchen]

partly closed by #1289 let us open new thread for further updates