Design a public Python API for the hparams plugin

To use the hparams dashboard, users currently have to manually construct hparams-specific protocol buffers and send them to file writers (see the tutorial notebook for an example; it takes a few dozen lines of Python code). The protobuf bindings are not particularly idiomatic Python, and are less than pleasant to use. We should investigate possible simplifications to this API.

For example, we could streamline the construction of the ListValues for the discrete domains by allowing the user to pass Python lists, and we can also infer the data types from the types of the elements of the domain* (which also lets us require that the list is homogeneously typed):

def create_experiment_summary():
  api = hparams_summary  # for brevity
  hparams = [
      api.hparam("num_units", domain=api.discrete([16, 32])),
      api.hparam("dropout_rate", domain=api.interval(min=0.1, max=0.2)),
      api.hparam("optimizer", domain=api.discrete(["adam", "sgd"])),
  ]
  metrics = [
      api.metric("accuracy"),
      api.metric("xent", display_name="cross-entropy"),
  ]
  experiment = api.experiment(hparams=hparams, metrics=metrics)
  experiment.write(logdir=os.path.join("logs", "hparam_tuning"))

This is just a sketch, but it’s already three times shorter than the current demo without (imho) any loss of utility.

* It’s fine to prohibit empty domains here. If a hyperparameter has empty domain, then the whole hyperparameter space is empty, so there can be no runs; thus, allowing empty domains is not actually useful.

[wchargin]

@moritzmeister: Long delay, I know, but: I’ve just added a trial_id kwarg to KerasCallback, hparams, and hparams_pb, as requested. The new forms are in latest nightly; see #2440/#2442.

I’m going to close this issue, since its original purpose has been completed and I don’t have any more planned changes in the works. Please feel free to open new issues for any further requests or feedback!

[wchargin]

A few people have expressed confusion about the term “session” as used by the current hparams API. In TensorFlow 1.x, tf.Session is a core piece of technical infrastructure for evaluating graphs. In the hparams API, a “session” is a single run of the model (training plus validation) with one set of hyperparameter values, but in TensorFlow 1.x this may correspond to many sess.run() calls, and in TensorFlow 2.x there are no sessions at all. The two notions of “session” are roughly unrelated.

I propose omitting “session” from new API symbols where feasible, and using “trial” instead. This is consistent with Vizier’s usage—from §1.2 of the Vizier paper,

A Trial is a list of parameter values, x, that will lead to a single evaluation of f(x).

—and, I think, also suggests the correct meaning.

“Session groups” would most literally become “trial groups”, though this name doesn’t make it obvious that these are specifically groups of trials with the same hyperparameters (nor did “session groups”). Rather than just using this literal replacement, we should try to convey the actual meaning: e.g., instead of asking for a “session group name”, ask for a “hparams key”.

Even better, though, I think that we can avoid asking for session group names at all in the common case. Rather than letting the session group name default to the session name, we should let it default to something like sha256(str(hparams)). This satisfies the intended behavior of session groups partitioning the trial space by hyperparameter values, without requiring any additional user input.