Automatic Hyperparameter Tuning in TFX

Per @zhitaoli’s suggestion a few days ago here, I am opening this up as a new thread.

I am wondering what is the best way to search for a model’s optimal hyperparameters within TFX.

It sounds like it may be possible to use the Trainer’s executor_class and specify an AI Platform Executor and even pass the additional arguments it needs for it to do hyperparameter tuning, but from Zhitao’s answer it sounds like receiving the responses may not be set up yet.

Anyway, I’m curious to hear what the roadmap to hyperparameter tuning within TFX looks like, and if there’s anything I can do to contribute to that effort.

[zhitaoli]

@andrewlarimer Some update: @1025KB has provided a custom component ‘tuner’ in upcoming release of TFX. Please see this example on how to use it.

For people who are interested, please feel free to try it out and provide some feedback.

Thanks.

[zhitaoli]

Sorry for the long delay as we have a couple of discussions with various teams (Google Cloud AI Platform, TFX internal, Keras, etc).

My current thought process on this one:

1. Given that Tensorflow is moving towards TF 2.0 and Keras, we hope to utilize this window to find a unified programming interface for hyper-parameter related business. The new keras-tuner project seems to provide a good starting point;
2. Once we agree on API to modeling work, different tuner implementations integrated with our pipeline system will be developed by collaboration between TFX::OSS and Cloud AI Platform/Katib/Keras, to realize hyper-parameter tuning on all thees platforms.
3. One important requirement is that the same model code and tuner component configuration should be portable as much as possible to different platform/environment. For instance, moving a locally loop-based tuner to GCP/Kubeflow should only require minimal new options which are only applicable to the specific environments.

@1025KB and I will publish more doc(s) and discussions in this thread as we make progress.