Failed to add jadmin to proxy and to spawn notebook

Bug description

Used the following method to install The Littlest JupyterHub on an Ubuntu 18.0404 LTS system:

curl https://raw.githubusercontent.com/jupyterhub/the-littlest-jupyterhub/master/bootstrap/bootstrap.py | sudo -E python3 - --admin jadmin

and the installation went fine, but, the spawner fails to start with the following error:

I am able to add users though, but, just the notebook does not get spawned.

Expected behaviour

The jadmin/ page to change the password should show up and the tree directory structure should be displayed.

Actual behaviour

The output of jouralctl -u jupyterhub.

-- Logs begin at Thu 2019-08-29 12:01:21 BST, end at Fri 2020-05-15 07:19:13 BST. --
May 15 06:28:05 foo systemd[1]: Started jupyterhub.service.
May 15 06:28:05 foo systemd[1]: Stopping jupyterhub.service...
May 15 06:28:05 foo systemd[1]: Stopped jupyterhub.service.
May 15 06:28:05 foo systemd[1]: Started jupyterhub.service.
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.305 JupyterHub app:2240] Running JupyterHub version 1.1.0
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.306 JupyterHub app:2271] Using Authenticator: firstuseauthenticator.firstuseauthenticator.FirstUseAuthenticator
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.307 JupyterHub app:2271] Using Spawner: builtins.UserCreatingSpawner
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.307 JupyterHub app:2271] Using Proxy: jupyterhub_traefik_proxy.toml.TraefikTomlProxy-0+unknown
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.317 JupyterHub app:1394] Writing cookie_secret to /opt/tljh/state/jupyterhub_cookie_secret
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.340 alembic.runtime.migration migration:155] Context impl SQLiteImpl.
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.340 alembic.runtime.migration migration:162] Will assume non-transactional DDL.
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.356 alembic.runtime.migration migration:515] Running stamp_revision  -> 4dc2d5a8c53c
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.369 alembic.runtime.migration migration:155] Context impl SQLiteImpl.
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.369 alembic.runtime.migration migration:162] Will assume non-transactional DDL.
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.436 JupyterHub app:1655] Not using whitelist. Any authenticated user will be allowed.
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.483 JupyterHub app:2311] Initialized 0 spawners in 0.010 seconds
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.486 JupyterHub app:2520] Not starting proxy
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.486 JupyterHub app:2556] Hub API listening on http://127.0.0.1:15001/hub/
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.486 JupyterHub app:2571] Starting managed service cull-idle
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.487 JupyterHub service:335] Starting service 'cull-idle': ['/opt/tljh/hub/bin/python3', '-m', 'jupyterhub_idle_culler', '--timeout=600', '--cull-every=60', '--concurrency=5', '--max-age=0']
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.489 JupyterHub service:121] Spawning /opt/tljh/hub/bin/python3 -m jupyterhub_idle_culler --timeout=600 --cull-every=60 --concurrency=5 --max-age=0
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.495 JupyterHub proxy:320] Checking routes
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.495 JupyterHub proxy:400] Adding default route for Hub: / => http://127.0.0.1:15001
May 15 06:28:07 foo python3[5859]: /opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_traefik_proxy/traefik_utils.py:42: RuntimeWarning: Escape character '_' cannot be a safe character. Set allow_collisions=True if you want to allow ambiguous escaped strings.
May 15 06:28:07 foo python3[5859]:   return server_type + "_" + escapism.escape(routespec, safe=safe)
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.517 JupyterHub proxy:129] Waiting for / to register with traefik
May 15 06:28:07 foo python3[5859]: [I 2020-05-15 06:28:07.730 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 18.25ms
May 15 06:28:08 foo python3[5859]: [I 2020-05-15 06:28:08.556 JupyterHub app:2631] JupyterHub is now running at http://:8000
May 15 06:29:07 foo python3[5859]: [I 2020-05-15 06:29:07.725 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 14.70ms
May 15 06:30:07 foo python3[5859]: [I 2020-05-15 06:30:07.725 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 14.49ms
May 15 06:31:07 foo python3[5859]: [I 2020-05-15 06:31:07.724 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 14.04ms
May 15 06:31:49 foo python3[5859]: [I 2020-05-15 06:31:49.596 JupyterHub log:174] 200 GET /hub/login (@127.0.0.1) 51.79ms
May 15 06:32:07 foo python3[5859]: [I 2020-05-15 06:32:07.723 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 13.19ms
May 15 06:32:14 foo python3[5859]: [I 2020-05-15 06:32:14.732 JupyterHub base:707] User logged in: jadmin
May 15 06:32:14 foo python3[5859]: [I 2020-05-15 06:32:14.735 JupyterHub log:174] 302 POST /hub/login?next= -> /hub/spawn (jadmin@127.0.0.1) 357.95ms
May 15 06:32:15 foo useradd[6178]: new group: name=jupyter-jadmin, GID=1013
May 15 06:32:15 foo useradd[6178]: new user: name=jupyter-jadmin, UID=1011, GID=1013, home=/home/jupyter-jadmin, shell=/bin/sh
May 15 06:32:15 foo python3[5859]: Adding user jupyter-jadmin to group jupyterhub-users
May 15 06:32:15 foo gpasswd[6185]: user jupyter-jadmin added by root to group jupyterhub-users
May 15 06:32:15 foo python3[5859]: Adding user jupyter-jadmin to group jupyterhub-admins
May 15 06:32:15 foo gpasswd[6189]: user jupyter-jadmin added by root to group jupyterhub-admins
May 15 06:32:15 foo python3[5859]: Running as unit: jupyter-jadmin.service
May 15 06:32:16 foo python3[5859]: [I 2020-05-15 06:32:16.013 JupyterHub log:174] 302 GET /hub/spawn -> /hub/spawn-pending/jadmin (jadmin@127.0.0.1) 1007.46ms
May 15 06:32:16 foo python3[5859]: [I 2020-05-15 06:32:16.290 JupyterHub pages:347] jadmin is pending spawn
May 15 06:32:16 foo python3[5859]: [I 2020-05-15 06:32:16.296 JupyterHub log:174] 200 GET /hub/spawn-pending/jadmin (jadmin@127.0.0.1) 14.35ms
May 15 06:32:16 foo python3[5859]: [I 2020-05-15 06:32:16.562 JupyterHub log:174] 200 GET /hub/api (@127.0.0.1) 2.05ms
May 15 06:32:16 foo python3[5859]: [I 2020-05-15 06:32:16.591 JupyterHub log:174] 200 POST /hub/api/users/jadmin/activity (jadmin@127.0.0.1) 20.54ms
May 15 06:32:16 foo python3[5859]: [W 2020-05-15 06:32:16.708 JupyterHub _version:56] jupyterhub version 1.1.0 != jupyterhub-singleuser version 1.0.0. This could cause failure to authenticate and result in redirect loops!
May 15 06:32:16 foo python3[5859]: [I 2020-05-15 06:32:16.708 JupyterHub base:855] User jadmin took 1.698 seconds to start
May 15 06:32:16 foo python3[5859]: [I 2020-05-15 06:32:16.709 JupyterHub proxy:262] Adding user jadmin to proxy /user/jadmin/ => http://127.0.0.1:50277
May 15 06:32:16 foo python3[5859]: /opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_traefik_proxy/traefik_utils.py:42: RuntimeWarning: Escape character '_' cannot be a safe character. Set allow_collisions=True if you want to allow ambiguous escaped strings.
May 15 06:32:16 foo python3[5859]:   return server_type + "_" + escapism.escape(routespec, safe=safe)
May 15 06:32:16 foo python3[5859]: [E 2020-05-15 06:32:16.711 JupyterHub base:876] Failed to add jadmin to proxy!
May 15 06:32:16 foo python3[5859]:     Traceback (most recent call last):
May 15 06:32:16 foo python3[5859]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 869, in finish_user_spawn
May 15 06:32:16 foo python3[5859]:         await self.proxy.add_user(user, server_name)
May 15 06:32:16 foo python3[5859]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub/proxy.py", line 274, in add_user
May 15 06:32:16 foo python3[5859]:         {'user': user.name, 'server_name': server_name},
May 15 06:32:16 foo python3[5859]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_traefik_proxy/toml.py", line 191, in add_route
May 15 06:32:16 foo python3[5859]:         self.toml_dynamic_config_file, self.routes_cache
May 15 06:32:16 foo python3[5859]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_traefik_proxy/traefik_utils.py", line 153, in persist_routes
May 15 06:32:16 foo python3[5859]:         toml.dump(routes_dict, config_fd)
May 15 06:32:16 foo python3[5859]:       File "/opt/tljh/hub/lib/python3.6/site-packages/toml/encoder.py", line 29, in dump
May 15 06:32:16 foo python3[5859]:         d = dumps(o, encoder=encoder)
May 15 06:32:16 foo python3[5859]:       File "/opt/tljh/hub/lib/python3.6/site-packages/toml/encoder.py", line 67, in dumps
May 15 06:32:16 foo python3[5859]:         raise ValueError("Circular reference detected")
May 15 06:32:16 foo python3[5859]:     ValueError: Circular reference detected
May 15 06:32:16 foo python3[5859]:     
May 15 06:32:16 foo python3[5859]: [E 2020-05-15 06:32:16.713 JupyterHub base:878] Stopping jadmin to avoid inconsistent state
May 15 06:32:17 foo python3[5859]: [W 2020-05-15 06:32:17.788 JupyterHub web:1786] 400 GET /hub/api/users/jadmin/server/progress (127.0.0.1): jadmin is not starting...
May 15 06:32:17 foo python3[5859]: [W 2020-05-15 06:32:17.789 JupyterHub log:174] 400 GET /hub/api/users/jadmin/server/progress (jadmin@127.0.0.1) 12.50ms
May 15 06:33:07 foo python3[5859]: [I 2020-05-15 06:33:07.724 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 14.06ms
May 15 06:33:08 foo python3[5859]: /opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_traefik_proxy/toml.py:104: RuntimeWarning: Escape character '_' cannot be a safe character. Set allow_collisions=True if you want to allow ambiguous escaped strings.
May 15 06:33:08 foo python3[5859]:   escaped_routespec = escapism.escape(traefik_routespec, safe=safe)
May 15 06:33:08 foo python3[5859]: [I 2020-05-15 06:33:08.561 JupyterHub proxy:320] Checking routes

How to reproduce

Just use the install script with curl to do an installation and try to open and login to the GUI.

Your personal set up

• OS: Ubuntu 18.04.4 LTS (x86_64)
• Version: JupyterHub 1.1.0

[GeorgianaElena]

Install toml 0.10.0:

/opt/tljh/hub/bin/python3 -m pip install toml==0.10.0

Restart hub and traefik services:

systemctl restart traefik
systemctl restart jupyterhub

@ilfsn, let me know if it works 🌞

[GeorgianaElena]

@apd47, I managed to reproduce this using Yuvi’s steps here. I think is has to do with the recent release of toml and not with the new escapism version as with the new version of escapism and the previous one of toml everything seems to be fine.

A temporary fix, until we figured out what’s the issue might be pinning the toml version to 0.10.0