Add support to log hparams and metrics to tensorboard?

How can I log metrics (e.g. validation loss of best epoch) together with the set of hyperparameters?

I have looked through the docs and through the code. It seems like an obvious thing, so maybe I’m just not getting it.

Currently, the only way that I found was to extend the logger class:

class MyTensorBoardLogger(TensorBoardLogger):

    def __init__(self, *args, **kwargs):
        super(MyTensorBoardLogger, self).__init__(*args, **kwargs)

    def log_hyperparams(self, *args, **kwargs):
        pass

    @rank_zero_only
    def log_hyperparams_metrics(self, params: dict, metrics: dict) -> None:
        params = self._convert_params(params)
        exp, ssi, sei = hparams(params, metrics)
        writer = self.experiment._get_file_writer()
        writer.add_summary(exp)
        writer.add_summary(ssi)
        writer.add_summary(sei)
        # some alternative should be added
        self.tags.update(params)

And then I’m writing the hparams with metrics in a callback:

def on_train_end(self, trainer, module):
        module.logger.log_hyperparams_metrics(module.hparams, {'val_loss': self.best_val_loss})

But that doesn’t seem right. Is there a better way to write some metric together with the hparams as well?

Environment

• OS: Ubuntu18.04
• conda4.8.3
• pytorch-lightning==0.7.1
• torch==1.4.0

[williamFalcon]

yes, let’s officially make this a fix! @mRcSchwering want to submit a PR?

[mRcSchwering]

Usually I am training and validating a model with different sets of hyperparameters in a loop. After each round the final output is often something like “val-loss”. This would be the validation loss of the best epoch achieved in this particular round. Eventually, I have a number of “best validation losses” and each of these represents a certain set of hyper parameters.

After the last training round I am looking at the various sets of hyperparameters and compare them to their associated best validation loss. Tensorboard already provides tools for that: Visualize the results in TensorBoard’s HParams plugin.

In your TensorBoardLogger you are already using the hparams function to summarize the hyperparameters. So, you are almost there. This function can also take metrics as a second argument. However, in your current implementation you always pass a {}. That’s why I had to overwrite your original implementation. Furthermore you are writing this summary once in the beginning of the round. But the metrics are only known at the very end.