katib-mysql: Access denied for user 'root'@'localhost' (using password: YES)

/kind bug

What steps did you take and what happened: Installed Kubeflow on an onpremise kubenetes cluster. Multiple Pods stuck in CrashLoopBackOff katib-db-manager katib-mysql

What did you expect to happen: Expect that the pods start normal

Anything else you would like to add: As mentioned in #1156 I increased liveness probe and readiness probe. I think, that the katib-db-manager also works if the katib-mysql is running.

kubectl -nkubeflow describe pod katib-mysql-98566c454-2kg4t
Name:         katib-mysql-98566c454-2kg4t
Namespace:    kubeflow
Priority:     0
Node:         
Start Time:   Fri, 08 Jan 2021 08:45:35 +0000
Labels:       app=katib
              app.kubernetes.io/component=katib
              app.kubernetes.io/name=katib-controller
              component=mysql
              pod-template-hash=98566c454
Annotations:  sidecar.istio.io/inject: false
Status:       Running
IP:           
IPs:
  IP:           
Controlled By:  ReplicaSet/katib-mysql-98566c454
Containers:
  katib-mysql:
    Container ID:  docker://8c14869f8a0cecdd6df1e6df2829838e11922167b66db1c6b5582d4ddbfb0818
    Image:         mysql:8
    Image ID:      docker-pullable://mysql@sha256:365e891b22abd3336d65baefc475b4a9a1e29a01a7b6b5be04367fcc9f373bb7
    Port:          3306/TCP
    Host Port:     0/TCP
    Args:
      --datadir
      /var/lib/mysql/datadir
    State:          Running
      Started:      Fri, 08 Jan 2021 08:45:37 +0000
    Ready:          False
    Restart Count:  0
    Liveness:       exec [/bin/bash -c mysqladmin ping -u root -p${MYSQL_ROOT_PASSWORD}] delay=30s timeout=5s period=10s #success=1 #failure=6
    Readiness:      exec [/bin/bash -c mysql -D ${MYSQL_DATABASE} -u root -p${MYSQL_ROOT_PASSWORD} -e 'SELECT 1'] delay=5s timeout=1s period=10s #success=1 #failure=6
    Environment:
      MYSQL_ROOT_PASSWORD:         <set to the key 'MYSQL_ROOT_PASSWORD' in secret 'katib-mysql-secrets'>  Optional: false
      MYSQL_ALLOW_EMPTY_PASSWORD:  true
      MYSQL_DATABASE:              katib
    Mounts:
      /var/lib/mysql from katib-mysql (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-5vvg8 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  katib-mysql:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  katib-mysql
    ReadOnly:   false
  default-token-5vvg8:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-5vvg8
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Normal   Scheduled  49s               default-scheduler  Successfully assigned kubeflow/katib-mysql-98566c454-2kg4t to 
  Normal   Pulled     47s               kubelet            Container image "mysql:8" already present on machine
  Normal   Created    47s               kubelet            Created container katib-mysql
  Normal   Started    47s               kubelet            Started container katib-mysql
  Warning  Unhealthy  3s (x4 over 33s)  kubelet            Readiness probe failed: mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

kubectl -nkubeflow logs katib-mysql-98566c454-2kg4t
2021-01-08 08:45:37+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.22-1debian10 started.
2021-01-08 08:45:38+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2021-01-08 08:45:38+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.22-1debian10 started.
2021-01-08T08:45:38.569814Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.22) starting as process 1
2021-01-08T08:45:38.676491Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2021-01-08T08:45:40.429366Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2021-01-08T08:45:40.636986Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2021-01-08T08:45:41.164747Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2021-01-08T08:45:41.164913Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2021-01-08T08:45:41.169798Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
2021-01-08T08:45:41.188249Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.22'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server - GPL.

kubectl -nkubeflow exec -it katib-mysql-98566c454-2kg4t -- /bin/bash
root@katib-mysql-98566c454-2kg4t:/# echo $MYSQL_ROOT_PASSWORD
test
root@katib-mysql-98566c454-2kg4t:/# echo $MYSQL_DATABASE
katib
root@katib-mysql-98566c454-2kg4t:/# mysqladmin ping -u root -p$MYSQL_ROOT_PASSWORD
mysqladmin: [Warning] Using a password on the command line interface can be insecure.
mysqladmin: connect to server at 'localhost' failed
error: 'Access denied for user 'root'@'localhost' (using password: YES)'
root@katib-mysql-98566c454-2kg4t:/# mysql -u root -e 'SELECT 1'
+---+
| 1 |
+---+
| 1 |
+---+
root@katib-mysql-98566c454-2kg4t:/# mysql -u root
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 41
Server version: 8.0.22 MySQL Community Server - GPL

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
4 rows in set (0.01 sec)

Environment:

• Kubeflow version: kfctl v1.2.0-0-gbc038f9
• OnPremise Kubernetes Cluster
• Kubernetes version: v1.17.0
• OS: Ubuntu 18.04.5 LTS

[martin-liu]

@chinmay22897 Yes.

It works after I exec to the mysql pod, and execute below commands:

mysql -u root -e "ALTER USER 'root'@'localhost' IDENTIFIED BY '$MYSQL_ROOT_PASSWORD'; RENAME USER 'root'@'localhost' to 'root'@'%'; create database if not exists katib;"

[vmarkovtsev]

Thank you @anneum your fix worked for me. This is what I did:

1. kubectl edit deployment katib-mysql -n kubeflow, then some vim-fu to update the delays.
2. I deleted the old database in my local storage, e.g. pvc-25dc2b81-9873-430f-8bc4-365fe5ff0357_kubeflow_katib-mysql.
3. kubectl delete pod -n kubeflow katib-mysql-...

Indeed, it took 2 minutes to initialize the DB.