Make TensorBoard aware of hyperparameters

Migrated from https://github.com/tensorflow/tensorflow/issues/7708.

@mmuneebs says:

I am wondering if it’s possible to store the arguments that we pass to the training script to be stored as part of other summaries, and have a way to see those arguments/parameters as notes in a separate tab in Tensorboard. Reason: It’s often hard to track individual training runs and relate them to the training/network config if you change training and network parameters frequently which always happens during hyperparameter search. We can do it manually too but if included in Tensorboard, it would make it one go-to visualizer and comparison tool for everything. One method of doing this that comes to mind is using the tf.app.flags.FLAGS arguments. This would keep everything standardized. Or we could also support argparse directly. Is this something in line with Tensorboard’s philosophy or is it too straightforward to be a special feature?

[koaning]

I should mention that I have little knowledge of the tensorboard/tensorflow internals and I was not aware of the database roadmap but to introduce the idea I had in mind I will first explain the hack we currently use.

Current Practice

We usually write command line apps with fire to run a bit of grid search from the command line. We can give the run a name from the command line and from the command line we also set any hyperparams (which optimiser, learning rate, dropout chance, etc). Something like:

$ tensoryo train --lr 0.001 --opt adam --dropout 20 --name 8-layers 

The command line app tensoryo will create a log folder over at <logdir>/8-layers-lr=0.001-opt=adam-dropout=20-id=<hash> and train a model.

Every run gets a randomised hash such that we can identify different id for each run.

Out current hack involves pushing the hyperparams into the log name and afterwards the grabbing any relevant data from the tflogs using the EventAccumulator. With a few loops and pandas we get what we want: a simple summary of all models where all the hyperparams are encoded in the logfolder name. It is a hack, but works very well for us.

The main usecase for these hyperparams though, at least for us, is to generate a large table of all the runs we’ve ever done in order to determine if perhaps there is a pattern. I imagine that this is the main usecase for most people: to be able to query summary statistics.

What Might Be Better

We are really just interested in getting a table with consistent summaries for every run. It would be great if tensorboard had a table view for something like:

id	dropout	lr	alg	acc_train	acc_val
8-layers-1	20	0.001	adam	0.97	0.96
8-layers-2	20	0.001	adam	0.94	0.95
8-layers-3	30	0.001	sgd	0.98	0.94
8-layers-4	30	0.001	sgd	0.99	0.93

You should be very easily be able to add support for this by having a summary method for it. This might look something like:

tf.summary.attribute('id', name + str(uuid()))
tf.summary.attribute('dropout`, 20)
tf.summary.attribute('lr', 0.001)
tf.summary.attribute('alg', stddev)
tf.summary.attribute('acc_train', training_loss.eval())
tf.summary.attribute('acc_val', validation_loss.eval())

Getting hyperparameters in general is certainly not trivial but I wonder if that is the feature developers really need. I can reason that developers would not mind to explicitly set these things manually as long as they are logged. Currently though, I am not aware of such a feature in tensorflow/tensorboard.

We’re looking into perhaps making a custom ui for this either way but it feels like a rather general feature for tensorboard that would adres the hyperparam issue. Am very curious to hear from you folks if this sounds sensible. It might very well be that my usecase is different from the general one.

[carlthome]

This is the best temporary workaround for a tf.estimator’s params dict that I could come up with, though it doesn’t let you compare runs. I would love something sortable like what @koaning describes.

hyperparameters = [tf.convert_to_tensor([k, str(v)]) for k, v in params.items()]
tf.summary.text('hyperparameters', tf.stack(hyperparameters))