Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Bayesian optimization experiment suggests same hyperparameters

See original GitHub issue

/kind bug

What steps did you take and what happened: [A clear and concise description of what the bug is.]

I’ve experienced the same hyperparameters being suggested by a Bayesian optimization experiment several times.

For example: with 1 parallel trial, and max trials of 4 the exact same hyperparameters are suggested for each trial

experiment_35

part of experiment yml file:

apiVersion: "kubeflow.org/v1alpha3"
kind: Experiment
metadata:
  namespace: kubeflow
  name: poi-experiment35
  labels:
    controller-tools.k8s.io: "1.0"
spec:
  objective:
    type: maximize
    goal: 0.99
    objectiveMetricName: fbeta_m
  metricsCollectorSpec:
    source:
      fileSystemPath:
        path: "/ml/models/abc/eval/"
        kind: Directory
    collector:
      kind: TensorFlowEvent
  algorithm:
    algorithmName: bayesianoptimization
    algorithmSettings:
      - name: "random_state"
        value: "5"
  parallelTrialCount: 1
  maxTrialCount: 4
  maxFailedTrialCount: 3
  parameters:
    - name: --tf_dense_size
      parameterType: int
      feasibleSpace:
        min: "128"
        max: "512"
    - name: --tf_dense_dropout_rate
      parameterType: double
      feasibleSpace:
        min: "0.01"
        max: "0.5"
    - name: --tf_learning_rate
      parameterType: double
      feasibleSpace:
        min: "0.00001"
        max: "0.001"
    - name: --tf_optimizer
      parameterType: categorical
      feasibleSpace:
        list:
        - sgd
        - rmsprop
        - adam

Another example: with 3 parallel trials, and 9 max trials, different hyperparameters are suggested for the first three trials, but for the following 6 trials the hyper-parameter combination from the best performing trial is suggested again for all six of the remaining trials.

part of experiment yml file:

apiVersion: "kubeflow.org/v1alpha3"
kind: Experiment
metadata:
  namespace: kubeflow
  name: poi-experiment36
  labels:
    controller-tools.k8s.io: "1.0"
spec:
  objective:
    type: maximize
    goal: 0.99
    objectiveMetricName: fbeta_m
  metricsCollectorSpec:
    source:
      fileSystemPath:
        path: "/ml/models/abc/eval/"
        kind: Directory
    collector:
      kind: TensorFlowEvent
  algorithm:
    algorithmName: bayesianoptimization
    algorithmSettings:
      - name: "random_state"
        value: "15"
  parallelTrialCount: 3
  maxTrialCount: 9
  maxFailedTrialCount: 3
  parameters:
    - name: --tf_dense_size
      parameterType: int
      feasibleSpace:
        min: "128"
        max: "512"
    - name: --tf_dense_dropout_rate
      parameterType: double
      feasibleSpace:
        min: "0.01"
        max: "0.5"
    - name: --tf_learning_rate
      parameterType: double
      feasibleSpace:
        min: "0.00001"
        max: "0.001"
    - name: --tf_optimizer
      parameterType: categorical
      feasibleSpace:
        list:
        - sgd
        - rmsprop
        - adam

What did you expect to happen: Different hyperparameters combinations to be suggested.

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.] I was just wondering if other people have experienced something similar.

Environment: Minikube version: N/A. Deployed on GCP.

Kubeflow version: kfctl v1.0-rc.1-0-g963c787
Kubernetes version: (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.8", GitCommit:"211047e9a1922595eaa3a1127ed365e9299a6c23", GitTreeState:"clean", BuildDate:"2019-10-15T12:11:03Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.8-gke.33", GitCommit:"2c6d0ee462cee7609113bf9e175c107599d5213f", GitTreeState:"clean", BuildDate:"2020-01-15T17:47:46Z", GoVersion:"go1.12.11b4", Compiler:"gc", Platform:"linux/amd64"}

OS: linux