TLS failure: `routines:CONNECT_CR_SRVR_HELLO:wrong version number`

Describe the bug For the life of me, I’m unable to get Ambassador to play nicely with SSL certs provided by cert-manager. Here’s the story so far:

1. Running ambassador 0.40.1 in a namespace called “ambassador”.

YAML config

# The namespace for all Ambassador related objects.
apiVersion: v1
kind: Namespace
metadata:
  name: ambassador
  labels:
    app: ambassador-proxy
---
# The admin service.
apiVersion: v1
kind: Service
metadata:
  labels:
    service: ambassador-admin
  name: ambassador-admin
  namespace: ambassador
spec:
  type: NodePort
  ports:
    - name: ambassador-admin
      port: 8877
      targetPort: 8877
  selector:
    service: ambassador
---
# The ClusterRole that specifies behaviors that the Ambassador Pods require.
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: ambassador
  namespace: ambassador
rules:
  - apiGroups: [""]
    resources:
      - services
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources:
      - configmaps
    verbs: ["create", "update", "patch", "get", "list", "watch"]
  - apiGroups: [""]
    resources:
      - secrets
    verbs: ["get", "list", "watch"]
---
# ServiceAccount that is mounted on each of the Ambassador Pods.
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ambassador
  namespace: ambassador
---
# Binding between the ClusterRole and ServiceAccout.
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: ambassador
  namespace: ambassador
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ambassador
subjects:
  - kind: ServiceAccount
    name: ambassador
    namespace: ambassador
---
# Deployment for the actual Ambassador Service.
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: ambassador
  namespace: ambassador
spec:
  replicas: 1
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"
      labels:
        service: ambassador
    spec:
      # This is where that ServiceAccount is declared for the Pods.
      serviceAccountName: ambassador
      containers:
        - name: ambassador
          image: quay.io/datawire/ambassador:0.40.1
          resources:
            limits:
              cpu: 1
              memory: 400Mi
            requests:
              cpu: 200m
              memory: 100Mi
          env:
            - name: AMBASSADOR_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - name: http
              containerPort: 80
            - name: https
              containerPort: 443
            - name: admin
              containerPort: 8877
          livenessProbe:
            httpGet:
              path: /ambassador/v0/check_alive
              port: 8877
            initialDelaySeconds: 30
            periodSeconds: 3
          readinessProbe:
            httpGet:
              path: /ambassador/v0/check_ready
              port: 8877
            initialDelaySeconds: 30
            periodSeconds: 3
      restartPolicy: Always
---
# Finally, the Ambassador Service.
apiVersion: v1
kind: Service
metadata:
  name: ambassador
  namespace: ambassador
  annotations:
    getambassador.io/config: |
      # From https://www.getambassador.io/user-guide/tls-termination/.
      apiVersion: ambassador/v0
      kind: Module
      name: tls
      config:
        # The 'server' block configures TLS termination. 'enabled' is the only
        # required element.
        server:
          # If 'enabled' is not True, TLS termination will not happen.
          enabled: True
          # If you set 'redirect_cleartext_from' to a port number, HTTP traffic
          # to that port will be redirected to HTTPS traffic. Typically you would
          # use port 80, of course.
          # redirect_cleartext_from: 80

          # These are optional. They should not be present unless you are using
          # a custom Docker build to install certificates onto the container
          # filesystem, in which case YOU WILL STILL NEED TO SET enabled: True
          # above.
          #
          # cert_chain_file: /etc/certs/tls.crt   # remember to set enabled!
          # private_key_file: /etc/certs/tls.key  # remember to set enabled!
          # Enable TLS ALPN protocol, typically HTTP2 to negotiate it with
          # HTTP2 clients over TLS.
          # This must be set to be able to use grpc over TLS.
          # alpn_protocols: h2
        # The 'client' block configures TLS client-certificate authentication.
        # 'enabled' is the only required element.
        client:
          # If 'enabled' is not True, TLS client-certificate authentication will
          # not happen.
          enabled: False

          # If 'cert_required' is True, TLS client certificates will be required
          # for every connection.
          # cert_required: False
          # This is optional. It should not be present unless you are using
          # a custom Docker build to install certificates onto the container
          # filesystem, in which case YOU WILL STILL NEED TO SET enabled: True
          # above.
          #
          # cacert_chain_file: /etc/cacert/tls.crt  # remember to set enabled!
spec:
  type: LoadBalancer
  # This must be a static IP in the same GCP region as the cluster.
  loadBalancerIP: <redacted>
  ports:
    - name: http
      protocol: TCP
      port: 80
    - name: https
      protocol: TCP
      port: 443
  selector:
    service: ambassador

1. cert-manager is running in its own namespace, “cert-manager”:

YAML config

# Source: cert-manager/templates/00-namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: "cert-manager"
  labels:
    name: "cert-manager"
    certmanager.k8s.io/disable-validation: "true"

---
# Source: cert-manager/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cert-manager
  namespace: "cert-manager"
  labels:
    app: cert-manager
    chart: cert-manager-v0.6.0-dev.6
    release: cert-manager
    heritage: Tiller
---
# Source: cert-manager/templates/certificate-crd.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: certificates.certmanager.k8s.io
  annotations:
    "helm.sh/hook": crd-install
  labels:
    app: cert-manager
    chart: cert-manager-v0.6.0-dev.6
    release: cert-manager
    heritage: Tiller
spec:
  group: certmanager.k8s.io
  version: v1alpha1
  scope: Namespaced
  names:
    kind: Certificate
    plural: certificates
    shortNames:
      - cert
      - certs

---
# Source: cert-manager/templates/challenge-crd.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: challenges.certmanager.k8s.io
  labels:
    app: cert-manager
    chart: cert-manager-v0.6.0-dev.6
    release: cert-manager
    heritage: Tiller
spec:
  group: certmanager.k8s.io
  version: v1alpha1
  names:
    kind: Challenge
    plural: challenges
  scope: Namespaced
---
# Source: cert-manager/templates/clusterissuer-crd.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: clusterissuers.certmanager.k8s.io
  annotations:
    "helm.sh/hook": crd-install
  labels:
    app: cert-manager
    chart: cert-manager-v0.6.0-dev.6
    release: cert-manager
    heritage: Tiller
spec:
  group: certmanager.k8s.io
  version: v1alpha1
  names:
    kind: ClusterIssuer
    plural: clusterissuers
  scope: Cluster
---
# Source: cert-manager/templates/issuer-crd.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: issuers.certmanager.k8s.io
  annotations:
    "helm.sh/hook": crd-install
  labels:
    app: cert-manager
    chart: cert-manager-v0.6.0-dev.6
    release: cert-manager
    heritage: Tiller
spec:
  group: certmanager.k8s.io
  version: v1alpha1
  names:
    kind: Issuer
    plural: issuers
  scope: Namespaced
---
# Source: cert-manager/templates/order-crd.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: orders.certmanager.k8s.io
  labels:
    app: cert-manager
    chart: cert-manager-v0.6.0-dev.6
    release: cert-manager
    heritage: Tiller
spec:
  group: certmanager.k8s.io
  version: v1alpha1
  names:
    kind: Order
    plural: orders
  scope: Namespaced
---
# Source: cert-manager/templates/rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: cert-manager
  labels:
    app: cert-manager
    chart: cert-manager-v0.6.0-dev.6
    release: cert-manager
    heritage: Tiller
rules:
  - apiGroups: ["certmanager.k8s.io"]
    resources:
      ["certificates", "issuers", "clusterissuers", "orders", "challenges"]
    verbs: ["*"]
  - apiGroups: [""]
    resources: ["configmaps", "secrets", "events", "services", "pods"]
    verbs: ["*"]
  - apiGroups: ["extensions"]
    resources: ["ingresses"]
    verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: cert-manager
  labels:
    app: cert-manager
    chart: cert-manager-v0.6.0-dev.6
    release: cert-manager
    heritage: Tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cert-manager
subjects:
  - name: cert-manager
    namespace: "cert-manager"
    kind: ServiceAccount
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cert-manager-view
  labels:
    app: cert-manager
    chart: cert-manager-v0.6.0-dev.6
    release: cert-manager
    heritage: Tiller
    rbac.authorization.k8s.io/aggregate-to-view: "true"
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
rules:
  - apiGroups: ["certmanager.k8s.io"]
    resources: ["certificates", "issuers"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cert-manager-edit
  labels:
    app: cert-manager
    chart: cert-manager-v0.6.0-dev.6
    release: cert-manager
    heritage: Tiller
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
rules:
  - apiGroups: ["certmanager.k8s.io"]
    resources: ["certificates", "issuers"]
    verbs: ["create", "delete", "deletecollection", "patch", "update"]
---
# Source: cert-manager/templates/deployment.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: cert-manager
  namespace: "cert-manager"
  labels:
    app: cert-manager
    chart: cert-manager-v0.6.0-dev.6
    release: cert-manager
    heritage: Tiller
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cert-manager
      release: cert-manager
  template:
    metadata:
      labels:
        app: cert-manager
        release: cert-manager
      annotations:
    spec:
      serviceAccountName: cert-manager
      containers:
        - name: cert-manager
          image: "quay.io/jetstack/cert-manager-controller:canary"
          imagePullPolicy: Always
          args:
            - --cluster-resource-namespace=$(POD_NAMESPACE)
            - --leader-election-namespace=$(POD_NAMESPACE)
          env:
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          resources:
            requests:
              cpu: 10m
              memory: 32Mi

1. A ClusterIssuer is created, answering DNS challenges:

YAML config

apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: <my email>
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-prod
    dns01:
      providers:
        - name: prod-clouddns
          clouddns:
            project: <redacted>
            serviceAccountSecretRef:
              name: prod-clouddns-svc-acct-secret
              key: cert-manager-svc-acct.json

1. A Certificate for cert-manager to establish:

YAML config

# See https://www.getambassador.io/user-guide/tls-termination/.
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: foo-cert
spec:
  secretName: ambassador-certs
  issuerRef:
    name: letsencrypt-staging
    kind: ClusterIssuer
  commonName: foo.kumo.run
  dnsNames:
    - foo.kumo.run
  acme:
    config:
      - dns01:
          provider: prod-clouddns
        domains:
          - foo.kumo.run

I can see from the cert-manager logs that the actual SSL certificate is acquired successfully and a Secret named ambassador-certs is created with the appropriate contents. However, when I try to execute an https query against Ambassador it’s not happy:

~ $ curl https://foo.kumo.run -v
* Rebuilt URL to: https://foo.kumo.run/
*   Trying 35.236.77.11...
* TCP_NODELAY set
* Connected to foo.kumo.run (35.236.77.11) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number
* stopped the pause stream!
* Closing connection 0
curl: (35) error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number

And yet nothing shows up in the Ambassador logs. I’ve tried restarting the Ambassador service in order to encourage it to pick up on the new certificate, but no matter what I try I keep getting the same exact error. It doesn’t matter if it’s a certificate from the staging Let’s Encrypt server or from prod.

It may also be worth noting that http requests to port 443 still work:

~ -> curl http://foo.kumo.run:443
Hello, world!
Version: 1.0.0
Hostname: foo-deployment-667cf67d6-92wnv

To Reproduce Steps to reproduce the behavior: everything explained above.

Expected behavior Ambassador to pick up on the cert in ambassador-certs as specified and respond correctly to https queries.

Versions (please complete the following information):

• Ambassador: 0.40.1
• Kubernetes environment: GKE

Additional context Documentation around TLS termination leaves a bit to be desired. For example, section 4 here (https://www.getambassador.io/user-guide/tls-termination/) indicates that “sometimes” you’ll need this extra config in order to make things work. Well why sometimes? What are those times? In what situations should that be necessary or unnecessary?

[samuela]

I created #1002 to make some clarifying changes to the docs in light of this discussion, so I’ll close this issue for now.

[idealhack]

@nbkrause Thanks for your help!

Another question, Ambassador document suggest adding the annotation on the ambassador service, but what about we have multiple certificates? I guess the annotations should be added to the services which need TLS?