Support Istio-sidecar handling

/kind feature

Describe the solution you’d like [A clear and concise description of what you want to happen.]

I’d like to run katib-experiment with istio-sidecar injection. I guess the reason why katib induce user to run katib-experiment with istio-sidecar injection “false” was something like this:

• https://github.com/kubeflow/katib/issues/955
• https://github.com/kubernetes/kubernetes/issues/65502
• https://github.com/istio/istio/issues/6324
• https://medium.com/redbox-techblog/handling-istio-sidecars-in-kubernetes-jobs-c392661c4af7

Thus, the following problem is maybe the main issue. A) katib’s metrics-logger-and-collector sidecar container should know when the istio-proxy sidecar container started B) metrics-logger-and-collector sidecar container should kill when the main container finished well or not.

---

By the way I think changing initial-logic and end-logic in metrics-collector image, katib could handle this.

Maybe something like this;

1. Handling issue B above from metrics-collector (https://github.com/kubeflow/katib/blob/eea6ada974ee11daf13fc66627696c458b4bad77/cmd/metricscollector/v1beta1/file-metricscollector/main.go#L339)

func killIstioProxySidecar() {
	klog.Infof("Let's kill istio-proxy sidecar")

	// curl -fsI -X POST http://localhost:15020/quitquitquit
	resp, err := http.Post("http://localhost:15020/quitquitquit", "text/plain", bytes.NewBufferString(""))
	if err != nil {
		// TODO 
		klog.Infof("there is no istio-proxy sidecar")
	}

	defer resp.Body.Close()
	klog.Infof("istio-proxy sidecar killed")
}

func main() {
	defer killIstioProxySidecar()

	flag.Var(&stopRules, "stop-rule", "The list of early stopping stop rules")
	flag.Parse()
	klog.Infof("Trial Name: %s", *trialName)
.....

1. Also, pass the process from istio-proxy sidecar nearby here

		// Ignore istio-sidecar processes
                // TODO this must be more sophisticated to catch istio-proxy's process
		if strings.Contains(cmdline, "envoy") || strings.Contains(cmdline, "sidecar") {
			fmt.Printf("Ignore istio-sidecar container's processes")
			continue
		}

1. Handling issue A above. I have no idea how to implement this now, but the primary container’s command should be changed automatically by katib experiment/trial’s controller as follows:

• Before

- command:
  - <YOUR_COMMAND>;

• After

- command:
  - /bin/sh
  - -c
  - |
    until curl -fsI http://localhost:15021/healthz/ready; do echo \"Waiting for Sidecar...\"; sleep 3; done;
    echo \"Sidecar available. Running the command...\";
    <YOUR_COMMAND>

• I guess there is the way since katib-controller already automatically adds echo completed > /var/log/katib/$$$$.pid at the end of primary-container’s command.

---

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

• I’m afraid this feature could be generalized except “Job” kind.
  • But for “Job”, this might be useful who wants to work with istio-sidecar

[anencore94]

Thanks @tenzen-y I changed the links

[longpi1]

I remember I solved this problem before, if you need, you can chat with me privately, or I will propose a PR。