Error accessing Hub API - HTTP 599: SSL certificate problem

Bug description

Binderhub instance launch fails because of SSL certificate issues for the jupyter hub.

Expected behaviour

Launch success.

Actual behaviour

Errors related to the jupyter hub access in the binder pod.

Error accessing Hub API (using <>/jupyter/hub/api/users/neurolibre-binder-tuto-8u2o82xt): HTTP 599: SSL certificate problem: unable to get local issuer certificate
Error creating user neurolibre-binder-tuto-8u2o82xt: HTTP 599: SSL certificate problem: unable to get local issuer certificate

What is weird is that we can access the binder webpage (through https, the cert is okay), and launch a build correctly.

How to reproduce

We are using a certificate from certbot, and manually inject it into k8s cluster: kubectl create secret tls <>-tls --namespace=binderhub --cert=certbot/fullchain.pem --key=certbot/privkey.pem

We are not using cert-manager because we have too much issues with it, and we will buy a certificate for our domain in the future.

Your personal set up

• OS: ubuntu 18
• Version: binder v0.2.0-n121.h6d936d7
• Configuration:

jupyterhub:
  ingress:
    enabled: true
    hosts:
      - <>
    annotations:
      kubernetes.io/ingress.class: nginx
      kubernetes.io/tls-acme: "true"
    https:
      enabled: true
      type: nginx
    config:
      # Allow POSTs of upto 64MB, for large notebook support.
      proxy-body-size: 64m
    tls:
      - secretName: <>-tls
        hosts:
          -<>
  hub:
    baseUrl: /jupyter/
    image:
      name: conpdev/hub-image
      tag: v1.4
  proxy:
    service:
      type: NodePort
  cull:
    timeout: 600
    every: 120
  singleuser:
    memory:
       guarantee: 2G
    cpu:
       guarantee: 1

# BinderHub config
config:
  GitHubRepoProvider:
    banned_specs:
      - ^(?!neurolibre\/.*).*
      - ^ines/spacy-binder.*
      - ^soft4voip/rak.*
      - ^hmharshit/cn-ait.*
      - ^shishirchoudharygic/mltraining.*
      - ^hmharshit/mltraining.*
  BinderHub:
    hub_url: https://<>/jupyter
    use_registry: true
    image_prefix: conpdev/binder-mcgill-

service:
  type: NodePort

storage:
  capacity: 2G

ingress:
  enabled: true
  hosts:
    - <>
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
  https:
    enabled: true
    type: nginx
  config:
    # Allow POSTs of upto 64MB, for large notebook support.
    proxy-body-size: 64m
  tls:
    - secretName: <>-tls
      hosts:
        - <>

[ltetrel]

We can continue the discussion it in the discourse forum. I will close this issue, since it seems to be not related to binderhub, but rather networking configuration.

[meeseeksmachine]

This issue has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/error-accessing-hub-api-http-599-ssl-certificate-problem/5824/1