[Discussion] Inclusion PV for the MySQL in Katib standalone manifests

/kind feature

See discussion: https://github.com/kubeflow/katib/pull/1464#issuecomment-796876215. Currently we exclude PV for MySQL in katib-with-kubeflow installation and include PV in katib-standalone installation.

Our users faced with many problems when they are using custom volume provisioner: https://github.com/kubeflow/katib/issues/1415, https://github.com/kubeflow/katib/issues/1156, https://github.com/kubeflow/katib/issues/1212.

We should discuss what is the easiest and stable way for the users to install Katib.

/cc @anneum @hjkkhj123 @Utkagr

[heilerich]

What is the value we should specify for the initialDelaySeconds to cover such cases?

My preferred solution would be to add a startupProbe with the same command as the livenessProbe as per the aforementioned kubernetes docs. Maybe with periodSeconds: 15 and failureThreshold: 60. That should guarantee a safe container startup in almost any setup. Additionally, as opposed to just increasing the initialDelaySeconds, you don’t have to compromise on delaying the restart of containers that already initialised successfully, but crashed again afterwards. I can’t really come up with any downside to giving a database initialisation a generous timeframe to finish with this method.

Do we have any other PV type that satisfies any K8s environment, whether it is GCP, AWS, on-prem, etc. ?

I don’t think you should be including any PV in the default installation manifests. It is common practice to just include the PVC and rely on the dynamic provisioning process to handle the rest. Dynamic provisioning is available at all big cloud providers and comes with many on-premise distributions nowadays. In fact, I have not encountered a PV that was bundled per default in any software that I deployed on kubernetes for a long time (of course excluding situations where it is explicitly necessary like installing drivers through a DaemonSet or physically tying hardware configuration to a node).

To me, including a PV with a specific storage flavour feels like actively circumventing the storage considerations that the administrators made for their environment. Which is fine if it is an optional setting, but might be dangerous if it is the default and can be overlooked when deploying katib.

[andreyvelich]

Thank you so much for sharing your thoughts @heilerich! Couple of questions:

I would suggest increasing the initialDelaySeconds or creating an additional startupProbe as per the official kubernetes docs

What is the value we should specify for the initialDelaySeconds to cover such cases?

You should seriously reconsider supplying a static hostPath PV in any circumstance.

Do we have any other PV type that satisfies any K8s environment, whether it is GCP, AWS, on-prem, etc. ?

In general, I agree with you @heilerich. Setup the proper PV is the part of Kubernetes infrastructure environment, not Katib responsibility.

We can provide the manifest for MySQL PV and don’t include it in the Kustomize install. In the documentation, we have to explain that user’s Kubernetes cluster should have the provisioner or user can deploy PV manifest.

What do you think @gaocegege @johnugeorge @yanniszark ?