katib-mysql does not work

/kind bug

What steps did you take and what happened: [A clear and concise description of what the bug is.]

i just install kubeflow in manual

What did you expect to happen:

2020-04-17 06:28:38+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.19-1debian10 started.
2020-04-17 06:28:38+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2020-04-17 06:28:38+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.19-1debian10 started.
2020-04-17T06:28:38.815249Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2020-04-17T06:28:38.815363Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.19) starting as process 1
mysqld: Table 'mysql.plugin' doesn't exist
2020-04-17T06:28:43.673845Z 0 [ERROR] [MY-010735] [Server] Could not open the mysql.plugin table. Please perform the MySQL upgrade procedure.
2020-04-17T06:28:44.007568Z 0 [Warning] [MY-010015] [Repl] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2020-04-17T06:28:44.391089Z 0 [Warning] [MY-010015] [Repl] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2020-04-17T06:28:44.509101Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2020-04-17T06:28:44.542561Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
2020-04-17T06:28:44.543088Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables
2020-04-17T06:28:44.543392Z 0 [ERROR] [MY-013129] [Server] A message intended for a client cannot be sent there as no client-session is attached. Therefore, we're sending the information to the error-log instead: MY-001146 - Table 'mysql.component' doesn't exist
2020-04-17T06:28:44.543560Z 0 [Warning] [MY-013129] [Server] A message intended for a client cannot be sent there as no client-session is attached. Therefore, we're sending the information to the error-log instead: MY-003543 - The mysql.component table is missing or has an incorrect definition.
2020-04-17T06:28:44.544225Z 0 [ERROR] [MY-010326] [Server] Fatal error: Can't open and lock privilege tables: Table 'mysql.user' doesn't exist
2020-04-17T06:28:44.544478Z 0 [ERROR] [MY-010952] [Server] The privilege system failed to initialize correctly. For complete instructions on how to upgrade MySQL to a new version please see the 'Upgrading MySQL' section from the MySQL manual.
2020-04-17T06:28:44.545053Z 0 [ERROR] [MY-010119] [Server] Aborting
2020-04-17T06:28:47.609375Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.19)  MySQL Community Server - GPL.

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

• Kubeflow version:
• Minikube version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

[hjkkhj123]

@andreyvelich Finally, I found solution this is just error with probe I just deploy simple mysql pod with rook/ceph PV it works so i change the probe interval and threshold it works fine now thank you for helping

[xwyangjshb]

我的情况不太一样，发现有两个版本mysql, 直接把8降低为5.7 ，

k3s kubectl edit deploy katib-mysql -nkubeflow
把image: mysql:8 ---> gcr.io/ml-pipeline/mysql

[root@dl01 manifests-1.3.0]# docker image ls |grep mysql
mysql                                                                           8                                                 76152be68449   10 days ago     524MB
gcr.io/ml-pipeline/mysql                                                        5.7                                               f8fcde8

``` 然后删除旧的pvc(katib-mysql)下的数据，删除katib-mysql， katib-db-manager pods, 自动重启pod，就恢复了正常。