internal_ssl + SlurmSpawner leads to certificate verification error

Bug description

I have setup a JupyterHub instance on my cluster’s login node that uses SlurmSpawner to spawn notebook servers on our cluster. I have verified that SlurmSpawner works (wonderfully btw) and that SSL works everywhere except between the Hub server and the spawned notebook servers. I was experimenting with JupyterHub’s internal_ssl feature but as soon as I set it to True in the config I was met with this error

[W 2020-09-19 20:15:21.818 SingleUserNotebookApp iostream:1432] SSL Error on 9 ('[IP]', 8081): [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1108)
[E 2020-09-19 20:15:21.819 SingleUserNotebookApp singleuser:434] Failed to connect to my Hub at https://[IP]:8081/hub/api (attempt 3/5). Is it running?
    Traceback (most recent call last):
      File "/opt/jupyterhub/lib/python3.8/site-packages/jupyterhub/singleuser.py", line 432, in check_hub_version
        resp = await client.fetch(self.hub_api_url)
      File "/opt/jupyterhub/lib/python3.8/site-packages/tornado/simple_httpclient.py", line 330, in run
        stream = await self.tcp_client.connect(
      File "/opt/jupyterhub/lib/python3.8/site-packages/tornado/tcpclient.py", line 293, in connect
        stream = await stream.start_tls(
      File "/opt/jupyterhub/lib/python3.8/site-packages/tornado/iostream.py", line 1417, in _do_ssl_handshake
        self.socket.do_handshake()
      File "/usr/lib/python3.8/ssl.py", line 1309, in do_handshake
        self._sslobj.do_handshake()
    ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1108

I have looked at #31, #103, and jupyterhub/jupyterhub#2055 but I cannot find good documentation on this issue/what I am doing wrong.

Your personal set up

JupyterHub instance using SlurmSpawner to spawn notebook servers. Hub instance is on the same machine as Slurm login node.

• OS: Ubuntu 20.04.1 LTS (all nodes)
• Version:

jupyter core     : 4.6.3
jupyter-notebook : 6.1.4
qtconsole        : not installed
ipython          : 7.18.1
ipykernel        : 5.3.4
jupyter client   : 6.1.7
jupyter lab      : 2.2.8
nbconvert        : 6.0.3
ipywidgets       : 7.5.1
nbformat         : 5.0.7
traitlets        : 5.0.4

• Configuration: jupyterhub.XXX.XXX is CNAME-d to a www server on our network and all traffic is proxied through the www server to the login node (where jupyterhub is hosted). infocube.XXX.XXX.XXX is the login node. jupyterhub_config.py: https://pastebin.com/BJRb3NfP

[leitec]

Hi @Hoeze, it’s been a while since I looked at this, and things may have changed since then.

I recall that the internal SSL mode uses fairly strict certificate validation. If the hub server is not in alt names, the singleuser process can’t provide the hub server with its address and port number. I think it’s expected that you will add the hub server hostname there. But then, if the node where singleuser is running isn’t listed in alt names, the hub server can’t contact the singleuser server at the given address and port.

This refers to the back end certificates created by JupyterHub for each session when internal_ssl is enabled, not the server certificate used on the user-facing JupyterHub endpoint, in case that’s what you meant by client.

[Hoeze]

I see, thanks @leitec!