[Proposal] Support PostHook for Experiment

/kind feature

Describe the solution you’d like [A clear and concise description of what you want to happen.]

I think there could be someone who would like to do something right after my experiments finished (succeeded or failed). For examples:

1. send slack notification with results
2. request some HTTP API (Delete data in minio except top K models)

Thus, I suggest to support this feature with postHook in Experiment CRD. (I’m not sure whether preHook will be necessary or not.)

Followings are my rough draft

type ExperimentSpec struct {
  StartTime *metav1.Time `json:"startTime,omitempty"`
  ...
  ...
  PostHooks []PostHookSpec `json:"postHooks,omitempty"`
}


type PostHookSpec struct {
  HookType PostHookType
  Job *batchv1.Job
}

type PostHookType string
const (
  OnSucceed PostHookType = "OnSucceeded"
  OnFailure PostHookType = "OnFailed"
}

I think supporting the main interface as k8s Job would be enough to handle any requests, but if there are another options, suggest them would be appreciated.

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Here are some references for support hooks in kubernetes :

• https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/
  • only Exec and HTTP is possible
• https://helm.sh/docs/topics/charts_hooks/
  • use Job to define hook
• https://argo-cd.readthedocs.io/en/stable/user-guide/resource_hooks/
  • use Job to define hook

[anencore94]

I think it’s better to handle out side of Katib, using event driven framework such as argo-events or knative-events. (I chose argo-events in our company). Maybe we can close this issue I think.

[gaocegege]

I am closing it. Feel free to re-open it if there is a new design.