Problem with `singleuser_image_spec`

Hi,

I am trying to setup a jupyterhub on our local network. All the machines are Ubuntu 16.04 and kubernetes seems to be setup properly. However, I have trouble spawning our custom built docker images in the kubernetes cluster.

To be precise, the following is my /etc/jupyterhub/config.py file:

c.JupyterHub.cookie_secret = b'bacd270d6831f754dff023509d154e5000d156c8d7f9b853f8b394947382534c'
from jupyter_client.localinterfaces import public_ips
c.JupyterHub.hub_ip = public_ips()[0]
c.JupyterHub.port = 443

c.JupyterHub.spawner_class = 'kubespawner.KubeSpawner'
c.KubeSpawner.singleuser_image_spec = # here a public image at dockerhub
c.KubeSpawner.cpu_limit = 2
c.KubeSpawner.mem_limit = '2G'

c.JupyterHub.ssl_cert = '/etc/letsencrypt/live/<domain>/fullchain.pem'
c.JupyterHub.ssl_key = '/etc/letsencrypt/live/<domain>/privkey.pem'

# ... some OAuthenticator and admin_user settings

If I comment out the singleuser_image_spec line, the default singleuser:latest image gets spawned. However, when I write the public image name, I get something along the lines:

[W 2018-06-27 19:58:54.920 JupyterHub base:698] User [e-mail] is slow to become responsive (timeout=10)
[I 2018-06-27 19:58:54.920 JupyterHub base:1003] [e-mail] is pending spawn
[I 2018-06-27 19:58:54.930 JupyterHub log:158] 200 GET /hub/user/[e-mail]/ ([e-mail]@::ffff:<ip_address>) 10114.97ms
[I 2018-06-27 19:59:05.192 JupyterHub log:158] 200 GET /hub/api (@192.168.0.5) 1.37ms
[W 2018-06-27 19:59:19.454 JupyterHub user:504] [e-mail]'s server never showed up at http://10.40.0.1:8888/user/[e-mail]/ after 30 seconds. Giving up
[E 2018-06-27 20:00:08.615 JupyterHub gen:940] Exception in Future <Task finished coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.5/dist-packages/jupyterhub/handlers/base.py:618> exception=TimeoutError("Server at http://10.40.0.1:8888/user/[e-mail]/ didn't respond in 30 seconds",)> after timeout
    Traceback (most recent call last):
      File "/usr/local/lib/python3.5/dist-packages/tornado/gen.py", line 936, in error_callback
        future.result()
      File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
        raise self._exception
      File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
        result = coro.send(None)
      File "/usr/local/lib/python3.5/dist-packages/jupyterhub/handlers/base.py", line 625, in finish_user_spawn
        await spawn_future
      File "/usr/local/lib/python3.5/dist-packages/jupyterhub/user.py", line 522, in spawn
        raise e
      File "/usr/local/lib/python3.5/dist-packages/jupyterhub/user.py", line 496, in spawn
        resp = await server.wait_up(http=True, timeout=spawner.http_timeout)
      File "/usr/local/lib/python3.5/dist-packages/jupyterhub/utils.py", line 197, in wait_for_http_server
        timeout=timeout
      File "/usr/local/lib/python3.5/dist-packages/jupyterhub/utils.py", line 155, in exponential_backoff
        raise TimeoutError(fail_message)
    TimeoutError: Server at http://10.40.0.1:8888/user/[e-mail]/ didn't respond in 30 seconds

By the way, in the worker nodes, I have already preinstalled the image via docker pull <image_name>.

Any opinions?

[consideRatio]

@aytekinar I’ve learned that only a few issues arise from breaking a JupyterHub version mismatch between the actual server and the user-images having it installed mainly for a startup command such as jupyterhub-singleuser (i think…).

I dont know the answers to your first question, Im very curious myself about it though. May depend on how the jupyterhub is configured also.

I recommend jupyter/minimal-notebook since it is something I know is verified to work well with jupyterhubs, the same goes for any image in the jupyter/docker-stacks repository. The jupyterhub/singleuser may work great, it probably does, I have not used it though. Sorry for the confusion!

Closing for now, feel free to reopen or create another issue if something more shows up!

[aytekinar]

@consideRatio thank you for your response.

Apparently, the key point was

• make sure to install jupyterhub of a matching version

Now docker images based on either jupyterhub/singleuser or jupyter/minimal-notebook work provided that I have the corresponding version of jupyterhub installed through pip. Maybe we can close this issue now, as I do not see any problem currently.

Just to understand why you recommend

I recommend using the minimal-notebook as a base for your custom images to avoid trouble.

I would like to ask the following:

• would any image that has jupyterhub installed suffice when it comes to spawning notebook servers over kubernetes or some other spawner,
• if so, why would you recommend jupyter/minimal-notebook then? For instance, jupyterhub/singleuser seems to occupy less space than jupyter/minimal-notebook.