jupyterhub/singleuser No user token cookie?

Hi, I run a docker swarm cluster based on mesos. I have configured my jupyterhub to authenticate with LDAP and then it starts the jupyter servers on swarm so I can see them appearing on the mesos cluster.

The problem is that those environments are unresponsive even when they seem ok. This is an example log:

I0421 14:12:49.394819 11264 exec.cpp:162] Version: 1.2.0
I0421 14:12:49.398325 11266 exec.cpp:237] Executor registered on agent b7cfea24-75c1-43d4-8727-49fb86ca66ae-S1
I0421 14:12:49.399019 11266 docker.cpp:850] Running docker -H unix:///var/run/docker.sock run --memory 2147483648 --env-file /tmp/XZKgjL -v /var/lib/mesos/slave/slaves/b7cfea24-75c1-43d4-8727-49fb86ca66ae-S1/frameworks/b2853a78-7cba-455a-99c7-d73bb5cbda95-0024/executors/jupyter-grodriguez.537d87935e09/runs/aa93541b-ab40-4903-a107-c137f1f8789b:/mnt/mesos/sandbox --net bridge --label=com.docker.swarm.mesos.name=jupyter-grodriguez --label=com.docker.swarm.mesos.task=jupyter-grodriguez.537d87935e09 --env=JPY_COOKIE_NAME=jupyter-hub-token-grodriguez --env=JPY_HUB_PREFIX=/hub/ --env=JPY_API_TOKEN=76bd38d982b947a9be0df68f06b1e721 --env=JUPYTERHUB_CLIENT_ID=user-grodriguez --env=JUPYTERHUB_HOST= --env=JPY_USER=grodriguez --env=JUPYTERHUB_API_TOKEN=76bd38d982b947a9be0df68f06b1e721 --env=MEM_LIMIT=2147483648 --env=JPY_BASE_URL=/user/grodriguez --env=JUPYTERHUB_OAUTH_CALLBACK_URL=/user/grodriguez/oauth_callback --env=JPY_HUB_API_URL=http://172.31.45.116:8081/hub/api --env=NOTEBOOK_DIR=/home/jovyan/work -p 80:8888/tcp --name mesos-b7cfea24-75c1-43d4-8727-49fb86ca66ae-S1.aa93541b-ab40-4903-a107-c137f1f8789b jupyterhub/singleuser
[W 2017-04-21 14:12:50.454 grodriguez configurable:168] Config option `open_browser` not recognized by `SingleUserNotebookApp`.  Did you mean `browser`?
[I 2017-04-21 14:12:50.459 grodriguez notebookapp:511] Writing notebook server cookie secret to /home/jovyan/.local/share/jupyter/runtime/notebook_cookie_secret
[W 2017-04-21 14:12:50.499 grodriguez login:222] All authentication is disabled.  Anyone who can connect to this server will be able to run code.
[I 2017-04-21 14:12:50.525 grodriguez extension:193] JupyterLab alpha preview extension loaded from /opt/conda/lib/python3.5/site-packages/jupyterlab
[I 2017-04-21 14:12:50.529 grodriguez notebookapp:1201] Serving notebooks from local directory: /home/jovyan/work
[I 2017-04-21 14:12:50.529 grodriguez notebookapp:1201] 0 active kernels 
[I 2017-04-21 14:12:50.529 grodriguez notebookapp:1201] The Jupyter Notebook is running at: http://0.0.0.0:8888/user/grodriguez/
[I 2017-04-21 14:12:50.529 grodriguez notebookapp:1202] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2017-04-21 14:12:50.546 grodriguez log:47] 302 GET /user/grodriguez (172.31.45.116) 0.63ms
[I 2017-04-21 14:12:50.947 grodriguez log:47] 302 GET /user/grodriguez (::ffff:192.168.16.108) 0.43ms
[I 2017-04-21 14:12:51.265 grodriguez log:47] 302 GET /user/grodriguez/tree? (::ffff:192.168.16.108) 0.81ms
[I 2017-04-21 14:12:52.068 grodriguez log:47] 302 GET /user/grodriguez/tree? (::ffff:192.168.16.108) 0.57ms
[I 2017-04-21 14:12:52.879 grodriguez log:47] 302 GET /user/grodriguez/tree? (::ffff:192.168.16.108) 0.58ms
[I 2017-04-21 14:12:53.746 grodriguez log:47] 302 GET /user/grodriguez/tree? (::ffff:192.168.16.108) 0.63ms
[I 2017-04-21 14:12:54.576 grodriguez log:47] 302 GET /user/grodriguez/tree? (::ffff:192.168.16.108) 0.60ms
[I 2017-04-21 14:12:55.328 grodriguez log:47] 302 GET /user/grodriguez/tree? (::ffff:192.168.16.108) 0.61ms
[I 2017-04-21 14:12:56.269 grodriguez log:47] 302 GET /user/grodriguez/tree? (::ffff:192.168.16.108) 0.56ms

Everything looks ok I think. However the server always responds with a 404 Not Found error and it keeps trying to redirect to the /user/grodriguez/tree page in an eternal loop.

So I enabled the debug (by passing --debug in the notebook command) and found this:

[I 2017-04-24 05:00:34.393 grodriguez notebookapp:511] Writing notebook server cookie secret to /home/jovyan/.local/share/jupyter/runtime/notebook_cookie_secret
[W 2017-04-24 05:00:34.431 grodriguez login:222] All authentication is disabled.  Anyone who can connect to this server will be able to run code.
[I 2017-04-24 05:00:34.442 grodriguez notebookapp:1201] Serving notebooks from local directory: /home/jovyan/work
[I 2017-04-24 05:00:34.442 grodriguez notebookapp:1201] 0 active kernels 
[I 2017-04-24 05:00:34.442 grodriguez notebookapp:1201] The Jupyter Notebook is running at: http://0.0.0.0:8888/user/grodriguez/
[I 2017-04-24 05:00:34.442 grodriguez notebookapp:1202] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2017-04-24 05:00:34.505 grodriguez log:47] 302 GET /user/grodriguez (172.31.45.116) 0.58ms
[I 2017-04-24 05:00:35.023 grodriguez log:47] 302 GET /user/grodriguez (::ffff:192.168.16.110) 0.37ms
[D 2017-04-24 05:00:35.261 grodriguez auth:219] No token cookie
[I 2017-04-24 05:00:35.261 grodriguez log:47] 302 GET /user/grodriguez/tree? (::ffff:192.168.16.110) 0.67ms
[D 2017-04-24 05:00:36.024 grodriguez auth:219] No token cookie
[I 2017-04-24 05:00:36.025 grodriguez log:47] 302 GET /user/grodriguez/tree? (::ffff:192.168.16.110) 0.76ms
[D 2017-04-24 05:00:36.767 grodriguez auth:219] No token cookie

So it is telling me the cookie is not found? But the cookie was created in the server:

cat /home/jovyan/.local/share/jupyter/runtime/notebook_cookie_secret                                                                                                          
UzrSzGx1qQ/sWL7lKdRN9JvgC6zBxiAa1KOp6qFVnB+y3/m+nRBt5ub/X+JfTugzxKlj8Jd7NZSQ
tkYjR9IMat+8Ic0MKRFCIEugJhvBE8VeSXOUUBQfUoYpVRwEIDW1Gws2DaZtG4bEo9d4f7drROg0
yPkaV52dBNLp1GETGSLNtWSSLHLu3scjpmkOAUCqdWDoTTatki5j8AzuU29gfieCA3Bmp23olrdg
aiIvAItJ9Cav+b+OnAoMNT+Jb1Vy6C4By5IGHPepnXRglkrdw51/Iak3RB8x/61rgJlvDqkNkceG
uLINnJAdmM5fyN70BVDsenlRu8Vhw59pjy15tjSqx0UCb1PcoqyuxTJuKxwVFAt04ELHqBXj3yrB
SE1PhHMsfvXLQIPdufK5+k+PfTyMWUcqOuG3d7Bz4co0j3IwZbuvh7+p0sUSkmbjIDuaeSezf0hx
0gxlXRhz0HZ9MbxDOpZQawc6+zYLk4rtZblhQyPawL+XC3MZFGWRjE6nTxF4aQY12mLDQ8V81UDb
kkANNjn23nzCq2pUlgSGuKNNdr/RFdrlx5O2yKQ+hDv1KRlMPaKzSOlQtmXV9cg4barTJjNnDEAh
PUHR/mYaO5EpzjQs/+0q/cJ8v+ByCvlvxb4KQu8FsXY3MHskabYX1+8kR/LeSRTiNVHDTOZga0Li
enG85FvCT371QG+/xi5yO/24PP1Kn2JIO76fF63eekfxT/aOZon3MDlsdd4EZhZY45aknW6Zy3DU
apt07teGl+jUXAvLoAD31qXNtKFOAW/qwkZr1pVH9bffT2GQiHt81g/FfdaGAJ5Hur1KwFJ1IB5l
CZgifdgvl1CSWKvo7fM9hKvMi9VJTns99SGgi5Q5PM/6C+M5CtOMAVFFWCPaxZG6Cq2pRk/eD86T
jU4QpGEaJgmlBMgmjEgtlgXo7ttAHKbxHizMItwkVT5xhZpZ0xPX1JOUARPojhjX0RW+fb+E9o6a
5BYQjdPHfeofSig4fWO69suLN2EVC8u4C7Tdezj6tSm24GF016Rs5z/ACjVOQVgdTl9F9ct6MIWi
qbKSqRRsRlrdHggNjyrkMDLIUtK1SsVrFKcf7Jq62tRNVBZwsOe40lFIOzZmdviECAItRVaJIBYI
o1bS/quY+7nM3BXbBBNhyBS+8B3A4xLeQs6YVcP+9TH4Zik54MfiKfr1hrT7m/JLVnECuBIj3e3Y
rr4oi00dZnnB5G0CXroocYTjdecGxZFMX3Y+PaNVQ0lHbj7FD/3vA2QNIY/0EyZIBu2ntKubg6cw
f0Cu646ROaYLEHj9WIMuh8l13baIXb+7EOy0gSqSwlOllMe7TaMh11gkvcwPU88YMihDBjwv/g==

I can only imagine that means is not in my browser? How do I get it into my browser then Any ideas would be appreciated.

Thanks!

[minrk]

Indeed, updating to master/0.8 requires updating both the Hub and the single-user server. Before the 0.8 release, I’ll try to make sure we get more informative messages about mismatched versions.

[mgeplf]

I believe that I also had the same problem: running a jupyterhub ‘server’ from master against a singleuser.py from a previous release (a conda installed version of jupyterhub (0.7.2)) causes the cookie lookup to fail, which in turn means the /tree/ path served by the singleuser.py tries to redirect to /hub/ and then an infinite redirection loop to happens.