Trials with same hyperparameters - Random search

/kind bug

What steps did you take and what happened: I had recently run 8 trials for an experiment and out of 8 only 4 unique combinations were seen and the rest were duplicates. The duplicates hyperparameters are seen in the majority of the experiments I run.
Ref: #842

Looking forward to this feature. #866

What did you expect to happen: Expected 8 different combinations of hyperparameter to run.

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.] Suggestion pod log:

INFO:pkg.suggestion.v1beta1.hyperopt.base_service:GetSuggestions returns 8 new Trial

Suggestion controller log: (I might have missed a few as many different experiments ran after this)

{"level":"info","ts":1624915551.6826324,"logger":"suggestion-controller","msg":"Sync assignments","Suggestion":"admin/test-123","Suggestion Requests":8,"Suggestion Count":8}
{"level":"info","ts":1624925599.2925694,"logger":"suggestion-controller","msg":"Sync assignments","Suggestion":"admin/test-123","Suggestion Requests":8,"Suggestion Count":8}
{"level":"info","ts":1624929816.4245157,"logger":"suggestion-controller","msg":"Sync assignments","Suggestion":"admin/test-123","Suggestion Requests":8,"Suggestion Count":8}
{"level":"info","ts":1624953809.892868,"logger":"suggestion-controller","msg":"Sync assignments","Suggestion":"admin/test-123","Suggestion Requests":8,"Suggestion Count":8}

Search space:

lr: 0.001
epochs: 100
batch size: 32, 64
num of cells: 40, 60
num of layers: 2, 3

Trail configuration:

Parallel trials: 8
Max trials: 8
Max failed trails: 2

Result:

This is all I can add as of now as the pods were cleaned up. If this isn’t enough please let me know what all I need to record when this happens again.

Environment:

• Kubeflow version (kfctl version): 1.3
• Kubernetes version: (use kubectl version): 1.19
• Katib controller version: v0.11.0

[johnugeorge]

I will try to reproduce this one. Thanks for reporting this

[johnugeorge]

related: #1633