trails succeeding but experiment hangs as metrics not collected

/kind bug

What steps did you take and what happened: created an experiment with stdout metrics collector, the trail container completes, i see the reqauired metrics printed to stdout, but it looks like the metrics collector sidecar is not injected and experiment hangs on the first n trails

What did you expect to happen: the trials would complete, the metrics would be collected and reported back , the next n trails would start

Anything else you would like to add:

Environment:

• Katib version (check the Katib controller image version): release-0.12
• Kubernetes version: (kubectl version):

Client Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.9-dispatcher", GitCommit:"2a8027f41d28b788b001389f3091c245cd0a9a60", GitTreeState:"clean", BuildDate:"2022-01-21T20:26:49Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.6-gke.1500", GitCommit:"7ce0f9f1939dfc1aee910732e84cba03840df91e", GitTreeState:"clean", BuildDate:"2021-11-17T09:30:26Z", GoVersion:"go1.16.9b7", Compiler:"gc", Platform:"linux/amd64"}

• OS (uname -a): Container-Optimized OS from Google

---

Impacted by this bug? Give it a 👍 We prioritize the issues with the most 👍

[andreyvelich]

@iantowey Try to specify imagePullPolicy: Always in the Katib controller and redeploy the controller. Maybe the image has been cached on your cluster.

I believe the defaulter webhook is working, because I can see metrics collector spec in your Experiment:

Metrics Collector Spec:
    Collector:
      Kind:  StdOut

[johnugeorge]

Since the reported issue is unrelated to Katib, closing this issue

/close