[tune] (KeyError: 'acc'): regression task: mse / mae as metric

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): CentOS Linux 7
• Ray installed from (source or binary): source
• Ray version: 0.7.3
• Python version: Python 3.6

Describe the problem

With Keras I wrote an neural network for a regression task based on this example (classification):

https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/tune_mnist_keras.py

The following procedure:

• Train function: neural network replaced
• Train function: metric: mse or mae specified

If the metric is mse or mae instead of acc, the following error message appears after execution:

self.reporter(keras_info=logs, mean_accuracy=logs[“acc”]) KeyError: ‘acc’

Are there other Keras examples specifically for regression tasks?

[richardliaw]

Try using this for the callback:

import keras
from ray.tune import track

class TuneReporterCallback(keras.callbacks.Callback):
    """Tune Callback for Keras."""

    def __init__(self, reporter=None, freq="batch", metrics=None, logs={}):
        """Initializer.
        Args:
            reporter (StatusReporter|tune.track.log|None): Tune object for
                returning results.
            freq (str): Sets the frequency of reporting intermediate results.
                One of ["batch", "epoch"].
        """
        self.reporter = reporter or track.log
        self.iteration = 0
        self.metrics = metrics or None
        if freq not in ["batch", "epoch"]:
            raise ValueError("{} not supported as a frequency.".format(freq))
        self.freq = freq
        super(TuneReporterCallback, self).__init__()

    def on_batch_end(self, batch, logs={}):
        if not self.freq == "batch":
            return
        self.iteration += 1
        for metric in list(logs):
            if "loss" in metric and "neg_" not in metric:
                logs["neg_" + metric] = -logs[metric]
        self.reporter(keras_info=logs, mean_accuracy=logs.get("acc"))

    def on_epoch_end(self, batch, logs={}):
        if not self.freq == "epoch":
            return
        self.iteration += 1
        for metric in list(logs):
            if "loss" in metric and "neg_" not in metric:
                logs["neg_" + metric] = -logs[metric]
        self.reporter(keras_info=logs, mean_accuracy=logs.get("acc"))

And for any place that you want to use a metric to optimize (i.e. AsyncHyperBandScheduler), you should instead replace it with: AsyncHyperBandScheduler(metric="keras_info/mse:, mode="min"..._

I’ll add more examples in upcoming days. Let me know if you run into anything - thanks!

[krfricke]

The Keras callback got updated in #10509. If the issue persist, please re-open with a reproducible script, and we will try to solve the problem.