ModelValidator does not work with regression model

I have created my own custom pipeline, using the taxi pipeline as basis to branch out from. So this pipeline contains everything from CsvExampleGen to the Pusher. I am trying to build a DNNLinearCombinedRegressor model. This all works fine until the ModelValidator component. It tries to compare the accuracy of the blessed model with the current model, however since we are running a regressor model, there is no accuracy metric and so the component fails on that point and with that the rest of the pipeline.

This is the error message that pops up here:

[2019-05-22 10:01:52,613] {logging_mixin.py:95} INFO - [2019-05-22 10:01:52,613] {executor.py:51} INFO - TF Logging: current metric: ((), {u’label/mean’: {‘doubleValue’: 5.047114372253418}, u’average_loss’: {‘doubleValue’: 335.8445129394531}, u’post_export_metrics/example_count’: {‘doubleValue’: 43299.0}, u’prediction/mean’: {‘doubleValue’: 5.458231449127197}}) [2019-05-22 10:01:52,613] {models.py:1788} ERROR - ‘accuracy’ Traceback (most recent call last): File “/home/username/taxi_pipeline/local/lib/python2.7/site-packages/airflow/models.py”, line 1657, in _run_raw_task result = task_copy.execute(context=context) File “/home/username/taxi_pipeline/local/lib/python2.7/site-packages/airflow/operators/python_operator.py”, line 103, in execute return_value = self.execute_callable() File “/home/username/taxi_pipeline/local/lib/python2.7/site-packages/airflow/operators/python_operator.py”, line 108, in execute_callable return self.python_callable(*self.op_args, **self.op_kwargs) File “/home/username/taxi_pipeline/local/lib/python2.7/site-packages/tfx/orchestration/airflow/airflow_adapter.py”, line 150, in python_exec executor.Do(self._input_dict, self._output_dict, self._exec_properties) File “/home/username/taxi_pipeline/local/lib/python2.7/site-packages/tfx/components/model_validator/executor.py”, line 164, in Do blessed_model_dir=blessed_model_dir) File “/home/username/taxi_pipeline/local/lib/python2.7/site-packages/tfx/components/model_validator/executor.py”, line 108, in _generate_blessing_result blessed_model_eval_result)): File “/home/username/taxi_pipeline/local/lib/python2.7/site-packages/tfx/components/model_validator/executor.py”, line 55, in _compare_eval_result if (current_metric[1][‘accuracy’][‘doubleValue’] < KeyError: ‘accuracy’

The code where this fails is in tfx/components/model_validator/executor.py in the _compare_eval_result definition:

  # TODO(jyzhao): customized validation support.
  def _compare_eval_result(self, current_model_eval_result,
                           blessed_model_eval_result):
    """Compare accuracy of all metrics and return true if current is better or equal."""
    for current_metric, blessed_metric in zip(
        current_model_eval_result.slicing_metrics,
        blessed_model_eval_result.slicing_metrics):
      # slicing_metric is a tuple, index 0 is slice, index 1 is its value.
      tf.logging.info("TF Logging: current metric: " + str(current_metric))
      tf.logging.info("TF Logging: current metric: " + str(blessed_metric))
      if current_metric[0] != blessed_metric[0]:
        raise RuntimeError('EvalResult not match {} vs {}.'.format(
            current_metric[0], blessed_metric[0]))
      if (current_metric[1]['accuracy']['doubleValue'] <
          blessed_metric[1]['accuracy']['doubleValue']):
        tf.logging.info(
            'Current model accuracy is worse than blessed model: {}'.format(
                current_metric[0]))
        return False
    return True

There seems to be no switch to check for accuracy when using a classification model and something else, like average_loss, when using a regressor model. I did a workaround by just replacing accuracy with average_loss, but this is hardcoded in the component code, so not very desirable.

Is there something that I am missing here, and is there actually a switch or a place to add custom configuration for the model validator? Or is this something that is indeed not implemented?

I am running in Python 2.7 and TFX 0.13.0.

Thanks!

[Efaq]

Also experiencing that. I wonder if there is some possible configuration to be passed to the validator. There is a (not so well) documented input called “eval_spec” in a documentation page but this doesn’t seem to be present in the actual code.

documentation page: https://www.tensorflow.org/tfx/guide/modelval

[Efaq]

@zhitaoli Ok, thank you for these updates!