x509: certificate relies on legacy Common Name field

/kind bug

What steps did you take and what happened: [A clear and concise description of what the bug is.]

1. Download release 0.10.0.
2. I installed Katib using the following commands. bash scripts/v1beta1/deploy.sh
3. I have started the sample experiment by running the following commands.

$ kubectl apply -f example/v1beta1/tpe-example.yaml
Error from server (InternalError): error when creating "/Users/tenzen/katib-0.10.0/examples/v1beta1/tpe-example.yaml": Internal error occurred: failed calling webhook "mutating.experiment.katib.kubeflow.org": Post "https://katib-controller.kubeflow.svc:443/mutate-experiments?timeout=30s": x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0

What did you expect to happen: I expected the experiment to start.

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

I found related topics in the release note for 1.19.0 https://kubernetes.io/docs/setup/release/notes/

Kubernetes is now built with golang 1.15.0-rc.1. The deprecated, legacy behavior of treating the CommonName field on X.509 serving certificates as a host name when no Subject Alternative Names are present is now disabled by default. It can be temporarily re-enabled by adding the value x509ignoreCN=0 to the GODEBUG environment variable. (#93264, @justaugustus) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scalability, Storage and Testing]

Environment:

• Kubeflow version (kfctl version): None
• Minikube version (minikube version): None
• Kubernetes version: (use kubectl version): 1.19.4
• OS (e.g. from /etc/os-release): ubuntu16.04(4.15.0-72-generic)

[trog-levrai]

Alright I figured things out for my case. If you’re trying to use openssl then you have to find a way to properly support the SAN field in the certificate. cfssl makes it easy but the example above wasn’t working for me with cfssl with version 1.5.0.

So here’s what worked for me with the kubeflow namespace (inspired from https://github.com/kelseyhightower/kubernetes-the-hard-way/issues/457#issuecomment-665581873).

Write the following files:

• ca-csr.json

{
    "CN": "katib-controller.kubeflow.svc",
    "hosts": [
        "katib-controller.kubeflow.svc"
    ],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "US",
            "ST": "CA",
            "L": "San Francisco"
        }
    ]
}

• server-csr.json (note there is no blankspace in the hosts value)

{
    "CN":"katib-controller.kubeflow.svc",
    "hosts":[
        "katib-controller.kubeflow.svc"
    ],
    "key":{
        "algo":"rsa",
        "size":2048
    },
    "names":[
        {
            "C":"US",
            "L":"CA",
            "ST":"San Francisco"
        }
    ]
}

• ca-config.json

{
    "signing": {
        "default": {
            "expiry": "168h"
        },
        "profiles": {
            "www": {
                "expiry": "8760h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth"
                ]
            },
            "client": {
                "expiry": "8760h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth"
                ]
            },
            "kubernetes": {
                "expiry": "876000h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                    ]
            }
        }
    }
}

From there you can run (note the kubernetes profile used):

# Generating SSL certs
cfssl gencert -initca ca-csr.json | cfssljson -bare ca –
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

# Needed renaming
mv ca.pem ca-cert.pem
mv server.pem cert.pem
mv server-key.pem key.pem

# K8S secret refresh
kubectl delete secret -n kubeflow katib-controller
kubectl create secret -n kubeflow generic katib-controller --from-file=ca-cert.pem --from-file=ca-key.pem --from-file=cert.pem --from-file=key.pem

# Check the katib-controller pod name and then
kubectl delete pod katib-controller-POD_ID -n kubeflow

[tenzen-y]

@zuiurs Thank you for the workaround suggestions! We succeeded in starting the experiment using the Secret created using the certificates generated by cfssl.

@andreyvelich Now I understand the cause of the error. And thank you for working on a solution to the problem.

For those who have encountered the same problem, here are the steps I took.

1. Create the certificates by cfssl. (Specify katib-controller.$NAMESPACE.svc in hosts.)
2. Remove Secret named katib-controller.
3. Create Secret using the following commands.

$ kubectl create secret -n $NAMESPACE generic katib-controller --from-file=ca-cert.pem --from-file=ca-key.pem --from-file=cert.pem --from-file=key.pem

1. Remove the katib-controller pod to mount the created Secret.