Invocation of post_start_cmd not part of debug output
See original GitHub issueBug description
I want to use the new post_start_cmd
and I am having some troubles with that. I hoped that a simple c.DockerSpawner.debug = True
would enable me to see what has gone wrong. However, I don’t see any additional helpful output.
Expected behaviour
I set the following post_start_cmd
so that scikit-learn databases could be downloaded before the participants arrive and start using the JupyterHub.
c.DockerSpawner.post_start_cmd = "cd /home/jovyan && jupyter nbconvert --execute --to notebook *.ipynb --stdout && touch .success
However, when I check the /home/jovyan
directory in the user’s Docker container, there is no file called .success
. Therefore I conclude that the Jupyter Notebook has not been executed. I would love to see the output of the post_start_cmd
somewhere in the Docker logs but I can’t.
A peak into https://docker-py.readthedocs.io/en/1.7.1/api/#exec_start shows that the return of that method should be some (potentially insightful?) text. Could this be written to the debug? As well as any other output created by the triggered Docker commands? Or would that level of debug output be too verbose and we would need a second level of verbosity here?
Actual behaviour
The logs remain silent about where my mistake resides. Each of the commands works inside the Docker container once it is started.
How to reproduce
S. a.
Your personal set up
- OS: Ubuntu 20.04
- JupyterHub: 1.3
- dockerspawner: 12.0
Jupyter Notebook Server Docker container logs
...
[I 2021-03-29 14:49:20.625 JupyterHub base:757] User logged in: user_xyp2tko8xi
[I 2021-03-29 14:49:20.628 JupyterHub log:181] 302 POST /hub/login?next= -> /hub/spawn (user_xyp2tko8xi@134.28.143.251) 38.07ms
[I 2021-03-29 14:49:20.714 JupyterHub dockerspawner:941] Container 'jupyter-user-5fxyp2tko8xi' is gone
[I 2021-03-29 14:49:20.779 JupyterHub dockerspawner:1162] Created container jupyter-user-5fxyp2tko8xi (id: ccc98e0) from image exam-scipy-notebook
[I 2021-03-29 14:49:20.779 JupyterHub dockerspawner:1186] Starting container jupyter-user-5fxyp2tko8xi (id: ccc98e0)
[I 2021-03-29 14:49:21.679 JupyterHub log:181] 302 GET /hub/spawn -> /hub/spawn-pending/user_xyp2tko8xi (user_xyp2tko8xi@134.28.143.251) 1011.49ms
[I 2021-03-29 14:49:21.720 JupyterHub pages:402] user_xyp2tko8xi is pending spawn
[I 2021-03-29 14:49:21.722 JupyterHub log:181] 200 GET /hub/spawn-pending/user_xyp2tko8xi (user_xyp2tko8xi@134.28.143.251) 4.79ms
[I 2021-03-29 14:49:22.918 JupyterHub log:181] 200 GET /hub/api (@172.20.0.5) 0.79ms
[I 2021-03-29 14:49:22.950 JupyterHub log:181] 200 POST /hub/api/users/user_xyp2tko8xi/activity (user_xyp2tko8xi@172.20.0.5) 22.77ms
[I 2021-03-29 14:49:23.048 JupyterHub base:904] User user_xyp2tko8xi took 2.370 seconds to start
[I 2021-03-29 14:49:23.049 JupyterHub proxy:257] Adding user user_xyp2tko8xi to proxy /user/user_xyp2tko8xi/ => http://172.20.0.5:8888
14:49:23.051 [ConfigProxy] info: Adding route /user/user_xyp2tko8xi -> http://172.20.0.5:8888
14:49:23.051 [ConfigProxy] info: Route added /user/user_xyp2tko8xi -> http://172.20.0.5:8888
14:49:23.052 [ConfigProxy] info: 201 POST /api/routes/user/user_xyp2tko8xi
[I 2021-03-29 14:49:23.054 JupyterHub users:664] Server user_xyp2tko8xi is ready
[I 2021-03-29 14:49:23.054 JupyterHub log:181] 200 GET /hub/api/users/user_xyp2tko8xi/server/progress (user_xyp2tko8xi@134.28.143.251) 1127.92ms
[I 2021-03-29 14:49:23.104 JupyterHub log:181] 302 GET /hub/spawn-pending/user_xyp2tko8xi -> /user/user_xyp2tko8xi/ (user_xyp2tko8xi@134.28.143.251) 4.73ms
[I 2021-03-29 14:49:23.252 JupyterHub log:181] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-user_xyp2tko8xi&redirect_uri=%2Fuser%2Fuser_xyp2tko8xi%2Foauth_callback&response_type=code&state=[secret] -> /user/user_xyp2tko8xi/oauth_callback?code=[secret]&state=[secret] (user_xyp2tko8xi@134.28.143.251) 17.76ms
[I 2021-03-29 14:49:23.332 JupyterHub log:181] 200 POST /hub/api/oauth2/token (user_xyp2tko8xi@172.20.0.5) 24.55ms
[I 2021-03-29 14:49:23.348 JupyterHub log:181] 200 GET /hub/api/authorizations/token/[secret] (user_xyp2tko8xi@172.20.0.5) 11.16ms
14:49:58.393 [ConfigProxy] info: 200 GET /api/routes
[I 2021-03-29 14:49:58.401 JupyterHub proxy:319] Checking routes
[I 2021-03-29 14:50:19.707 JupyterHub log:181] 200 POST /hub/api/users/admin/activity (admin@172.20.0.4) 11.28ms
[I 2021-03-29 14:52:49.117 JupyterHub log:181] 200 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-admin&redirect_uri=%2Fuser%2Fadmin%2Foauth_callback&response_type=code&state=[secret] (user_xyp2tko8xi@134.28.143.251) 21.48ms
[I 2021-03-29 14:52:49.158 JupyterHub log:181] 200 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-admin&redirect_uri=%2Fuser%2Fadmin%2Foauth_callback&response_type=code&state=[secret] (user_xyp2tko8xi@134.28.143.251) 5.77ms
[I 2021-03-29 14:52:49.648 JupyterHub log:181] 200 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-admin&redirect_uri=%2Fuser%2Fadmin%2Foauth_callback&response_type=code&state=[secret] (user_xyp2tko8xi@134.28.143.251) 6.45ms
[I 2021-03-29 14:52:49.694 JupyterHub log:181] 200 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-admin&redirect_uri=%2Fuser%2Fadmin%2Foauth_callback&response_type=code&state=[secret] (user_xyp2tko8xi@134.28.143.251) 11.64ms
...
The JupyterHub Docker container logs
[D 2021-03-29 14:45:18.683 SingleUserNotebookApp auth:874] Allowing Hub user admin
[D 2021-03-29 14:45:18.684 SingleUserNotebookApp log:181] 304 GET /user/admin/custom/custom.js?v=20210329144509 (admin@134.28.143.251) 1.68ms
[D 2021-03-29 14:45:18.729 SingleUserNotebookApp auth:874] Allowing Hub user admin
[I 2021-03-29 14:45:18.730 SingleUserNotebookApp log:181] 101 GET /user/admin/terminals/websocket/1 (admin@134.28.143.251) 1.58ms
[D 2021-03-29 14:45:18.731 SingleUserNotebookApp zmqhandlers:154] Opening websocket /user/admin/terminals/websocket/1
TermSocket.open: 1
TermSocket.open: Opened 1
Websocket closed
[D 2021-03-29 14:49:00.515 SingleUserNotebookApp auth:874] Allowing Hub user admin
[I 2021-03-29 14:49:00.517 SingleUserNotebookApp log:181] 302 GET /user/admin/logout -> /hub/logout (admin@134.28.143.251) 2.96ms
[D 2021-03-29 14:50:19.691 SingleUserNotebookApp mixins:499] Notifying Hub of activity 2021-03-29T14:48:58.680381Z
[D 2021-03-29 14:52:49.018 SingleUserNotebookApp auth:504] No user identified
[D 2021-03-29 14:52:49.021 SingleUserNotebookApp auth:835] Redirecting to login url: /hub/api/oauth2/authorize?client_id=jupyterhub-user-admin&redirect_uri=%2Fuser%2Fadmin%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogImQwMzkxYjU2NTE0ZDRiYmI4ZWY2MmE1ZTNmODNiOTM4IiwgIm5leHRfdXJsIjogIi91c2VyL2FkbWluL2FwaS9zZXNzaW9ucz9fPTE2MTcwMjkxMTIzNzUifQ
[I 2021-03-29 14:52:49.022 SingleUserNotebookApp log:181] 302 GET /user/admin/api/sessions?_=1617029112375 -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-admin&redirect_uri=%2Fuser%2Fadmin%2Foauth_callback&response_type=code&state=[secret] (@134.28.143.251) 4.82ms
[D 2021-03-29 14:52:49.057 SingleUserNotebookApp auth:504] No user identified
[D 2021-03-29 14:52:49.058 SingleUserNotebookApp auth:835] Redirecting to login url: /hub/api/oauth2/authorize?client_id=jupyterhub-user-admin&redirect_uri=%2Fuser%2Fadmin%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogImFiY2JjYzliZWM2NzQ0NWE4ZDk4OTQ5NThiZTExYjk0IiwgIm5leHRfdXJsIjogIi91c2VyL2FkbWluL2FwaS90ZXJtaW5hbHM_Xz0xNjE3MDI5MTEyMzc2In0
[I 2021-03-29 14:52:49.059 SingleUserNotebookApp log:181] 302 GET /user/admin/api/terminals?_=1617029112376 -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-admin&redirect_uri=%2Fuser%2Fadmin%2Foauth_callback&response_type=code&state=[secret] (@134.28.143.251) 2.23ms
[D 2021-03-29 14:52:49.570 SingleUserNotebookApp auth:504] No user identified
[W 2021-03-29 14:52:49.570 SingleUserNotebookApp auth:682] Detected unused OAuth state cookies
[D 2021-03-29 14:52:49.571 SingleUserNotebookApp auth:835] Redirecting to login url: /hub/api/oauth2/authorize?client_id=jupyterhub-user-admin&redirect_uri=%2Fuser%2Fadmin%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogImMwMzNhOGFlYmUwODQxOTA5ODgyODQ5NzE5MDJlZjlmIiwgIm5leHRfdXJsIjogIi91c2VyL2FkbWluL2FwaS9zZXNzaW9ucz9fPTE2MTcwMjgyNDIyMTMiLCAiY29va2llX25hbWUiOiAianVweXRlcmh1Yi11c2VyLWFkbWluLW9hdXRoLXN0YXRlLXRyQktERFFzIn0
[I 2021-03-29 14:52:49.571 SingleUserNotebookApp log:181] 302 GET /user/admin/api/sessions?_=1617028242213 -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-admin&redirect_uri=%2Fuser%2Fadmin%2Foauth_callback&response_type=code&state=[secret] (@134.28.143.251) 1.90ms
[D 2021-03-29 14:52:49.606 SingleUserNotebookApp auth:504] No user identified
[W 2021-03-29 14:52:49.607 SingleUserNotebookApp auth:682] Detected unused OAuth state cookies
[D 2021-03-29 14:52:49.607 SingleUserNotebookApp auth:835] Redirecting to login url: /hub/api/oauth2/authorize?client_id=jupyterhub-user-admin&redirect_uri=%2Fuser%2Fadmin%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogImExYzYwNzhmNjAxZjQzYmFhNTdhMDJmY2YxYzUwNDQ3IiwgIm5leHRfdXJsIjogIi91c2VyL2FkbWluL2FwaS90ZXJtaW5hbHM_Xz0xNjE3MDI4MjQyMjE0IiwgImNvb2tpZV9uYW1lIjogImp1cHl0ZXJodWItdXNlci1hZG1pbi1vYXV0aC1zdGF0ZS1EUndRWXRwbiJ9
[I 2021-03-29 14:52:49.608 SingleUserNotebookApp log:181] 302 GET /user/admin/api/terminals?_=1617028242214 -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-admin&redirect_uri=%2Fuser%2Fadmin%2Foauth_callback&response_type=code&state=[secret] (@134.28.143.251) 2.22ms
[D 2021-03-29 14:54:56.186 SingleUserNotebookApp mixins:499] Notifying Hub of activity 2021-03-29T14:48:58.680381Z
[D 2021-03-29 14:57:52.023 SingleUserNotebookApp auth:504] No user identified
[W 2021-03-29 14:57:52.023 SingleUserNotebookApp auth:682] Detected unused OAuth state cookies
[D 2021-03-29 14:57:52.024 SingleUserNotebookApp auth:835] Redirecting to login url: /hub/api/oauth2/authorize?client_id=jupyterhub-user-admin&redirect_uri=%2Fuser%2Fadmin%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogIjRkMTI4ZTY3MmFjYzQzY2I5N2Q3NzhmMzI1NGI5YjYyIiwgIm5leHRfdXJsIjogIi91c2VyL2FkbWluL2FwaS9zZXNzaW9ucz9fPTE2MTcwMjgyNDIyMTUiLCAiY29va2llX25hbWUiOiAianVweXRlcmh1Yi11c2VyLWFkbWluLW9hdXRoLXN0YXRlLUtydVhBRWZkIn0
Issue Analytics
- State:
- Created 2 years ago
- Comments:7 (2 by maintainers)
Top GitHub Comments
Thank you very much, @manics ! Your hint with
sh -c '...'
solved it for me!However, here are the logs that I would expect to contain some indication:
and
Now you might guess that in one cell of a Jupyter Notebook I added a
!wget https://google.com
, saved it and restarted my Jupyter Notebook server. But you can’t read it from the logs. However, my expections would be thatc.DockerSpawner.debug = True
would provide me with exactly that information.I am not sure how much should be logged from the Docker client in general. If in my case all stdout and stderr had been dumped to
DEBUG
, that would have been a great help! SoDEBUG
would be my first suggestion. If one decides to do the logging at the Docker client, I guess it should be placed somewhere close to https://github.com/jupyterhub/dockerspawner/blob/master/dockerspawner/dockerspawner.py#L903. But most of the times things work fine and it could create a lot of noise. Therefore, maybe a use-case specific handling at https://github.com/jupyterhub/dockerspawner/blob/master/dockerspawner/dockerspawner.py#L745 could be more straight-forward.Thanks, for me the pull request #427 solved this perfectly!