internal_ssl + SlurmSpawner leads to certificate verification error
See original GitHub issueBug description
I have setup a JupyterHub instance on my cluster’s login node that uses SlurmSpawner
to spawn notebook servers on our cluster. I have verified that SlurmSpawner
works (wonderfully btw) and that SSL works everywhere except between the Hub server and the spawned notebook servers. I was experimenting with JupyterHub’s internal_ssl
feature but as soon as I set it to True
in the config I was met with this error
[W 2020-09-19 20:15:21.818 SingleUserNotebookApp iostream:1432] SSL Error on 9 ('[IP]', 8081): [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1108)
[E 2020-09-19 20:15:21.819 SingleUserNotebookApp singleuser:434] Failed to connect to my Hub at https://[IP]:8081/hub/api (attempt 3/5). Is it running?
Traceback (most recent call last):
File "/opt/jupyterhub/lib/python3.8/site-packages/jupyterhub/singleuser.py", line 432, in check_hub_version
resp = await client.fetch(self.hub_api_url)
File "/opt/jupyterhub/lib/python3.8/site-packages/tornado/simple_httpclient.py", line 330, in run
stream = await self.tcp_client.connect(
File "/opt/jupyterhub/lib/python3.8/site-packages/tornado/tcpclient.py", line 293, in connect
stream = await stream.start_tls(
File "/opt/jupyterhub/lib/python3.8/site-packages/tornado/iostream.py", line 1417, in _do_ssl_handshake
self.socket.do_handshake()
File "/usr/lib/python3.8/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1108
I have looked at #31, #103, and jupyterhub/jupyterhub#2055 but I cannot find good documentation on this issue/what I am doing wrong.
Your personal set up
JupyterHub instance using SlurmSpawner
to spawn notebook servers. Hub instance is on the same machine as Slurm login node.
- OS: Ubuntu 20.04.1 LTS (all nodes)
- Version:
jupyter core : 4.6.3
jupyter-notebook : 6.1.4
qtconsole : not installed
ipython : 7.18.1
ipykernel : 5.3.4
jupyter client : 6.1.7
jupyter lab : 2.2.8
nbconvert : 6.0.3
ipywidgets : 7.5.1
nbformat : 5.0.7
traitlets : 5.0.4
- Configuration:
jupyterhub.XXX.XXX
is CNAME-d to a www server on our network and all traffic is proxied through the www server to the login node (where jupyterhub is hosted).infocube.XXX.XXX.XXX
is the login node.jupyterhub_config.py
: https://pastebin.com/BJRb3NfP
Issue Analytics
- State:
- Created 3 years ago
- Comments:5 (2 by maintainers)
Top Results From Across the Web
A Simple Explanation of SSL Certificate Errors & How to Fix ...
What is an SSL certificate error? An SSL certificate error occurs when a web browser can't verify the SSL certificate installed on a...
Read more >451 4.7.5 [internal] SSL cert must be signed by a valid CA
Cause. The TLS certificate on the remote MTA is failing the certificate verification. This may be due to an untrusted certificate authority in ......
Read more >Internal SSL Error is not enough information
"Internal SSL Error" means a secure connection could not be established, ... you are using does not match the FQDN of the SSL...
Read more >Policy installation fails with "Internal SSL authentication SSL ...
fwCA::RevokeCertsForCN: revoked certificates which were in PENDING status (DN "CN=NAME,O=OTHERNAME"). When resetting SIC it fails with: "Failed ...
Read more >Enable Oracle Analytics Server Internal SSL
Enable server checking of client certificates. Specify cipher suite to use. ... ORACLE_HOME/user_projects/domains/bi/bitools/bin/ssl.sh internalssl true.
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Hi @Hoeze, it’s been a while since I looked at this, and things may have changed since then.
I recall that the internal SSL mode uses fairly strict certificate validation. If the hub server is not in alt names, the singleuser process can’t provide the hub server with its address and port number. I think it’s expected that you will add the hub server hostname there. But then, if the node where singleuser is running isn’t listed in alt names, the hub server can’t contact the singleuser server at the given address and port.
This refers to the back end certificates created by JupyterHub for each session when internal_ssl is enabled, not the server certificate used on the user-facing JupyterHub endpoint, in case that’s what you meant by client.
I see, thanks @leitec!