Data-Aware Calibration

The current calibration algorithm use global_scale to estimate the scale of intermediate result, which is configured by user. For better accuracy loss, we need to implement data-aware calibration algorithm: given a small calibration dataset (like 100 samples), to achieve better scale estimation.

Implementation

Collecting Intermediate Results

To collect intermediate results of every operation in the graph, we may need to implement a Visitor to return every Call as output. The API would be: stats = collect_stats(graph, data)

Modification for SimulatedQuantize

It is kinds of hard to save the mapping from origin operator to the operator after annotation. So we may need to add an option for simulated_quantize, like mode='origin', which denotes that sq will not simulate the rounding error/saturated error, instead, just return the input directly. With this, we can collect the intermediate result of the original graph.

Calibration Procedure

Having the calibration data, we can adjust the scale according to the output of the annotated graph. It is actually an optimization problem. The target can be the KL divergence between outputs of original graph and annotated graph, and the adjustment method can be simply search-based or learning-based. There should be lots of room for exploration.

[eqy]

I am very interested in this problem, and I have a few different approaches I want to try out. However, I wonder if requiring intermediate results to be collected is too labor intensive or unnecessary. In the AutoTVM setting, we use domain specific features as they are powerful and allow for transfer learning. However, transfer learning is not as important in this setting, and it could be that blackbox features (e.g., just the scale configuration) may be enough to build a usable machine learning model (just as knob features are usable in AutoTVM). I think if that turns out to already work then we can save a significant amount of effort. It seems slightly expensive to compute both the original graph and annotated graph every time we want to do try a different calibration configuration, so I think it would be great if we could get away with doing even less work 😃.

I’ll send updates on this approach as I get results. You may have already guessed that performance is important as I am first tuning the workloads required for quantization so trying many calibration configurations is not too slow.

[tqchen]

closed by #3538 let us open new thread for tutorials