Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Invocation of post_start_cmd not part of debug output
See original GitHub issueBug description
I want to use the new post_start_cmd and I am having some troubles with that. I hoped that a simple c.DockerSpawner.debug = True would enable me to see what has gone wrong. However, I don’t see any additional helpful output.
Expected behaviour
I set the following post_start_cmd so that scikit-learn databases could be downloaded before the participants arrive and start using the JupyterHub.
c.DockerSpawner.post_start_cmd = "cd /home/jovyan && jupyter nbconvert --execute --to notebook *.ipynb --stdout && touch .success
However, when I check the /home/jovyan directory in the user’s Docker container, there is no file called .success. Therefore I conclude that the Jupyter Notebook has not been executed. I would love to see the output of the post_start_cmd somewhere in the Docker logs but I can’t.
A peak into https://docker-py.readthedocs.io/en/1.7.1/api/#exec_start shows that the return of that method should be some (potentially insightful?) text. Could this be written to the debug? As well as any other output created by the triggered Docker commands? Or would that level of debug output be too verbose and we would need a second level of verbosity here?
Actual behaviour
The logs remain silent about where my mistake resides. Each of the commands works inside the Docker container once it is started.
How to reproduce
S. a.
Your personal set up
- OS: Ubuntu 20.04
- JupyterHub: 1.3
- dockerspawner: 12.0
Jupyter Notebook Server Docker container logs
...
[I 2021-03-29 14:49:20.625 JupyterHub base:757] User logged in: user_xyp2tko8xi
[I 2021-03-29 14:49:20.628 JupyterHub log:181] 302 POST /hub/login?next= -> /hub/spawn (user_xyp2tko8xi@134.28.143.251) 38.07ms
[I 2021-03-29 14:49:20.714 JupyterHub dockerspawner:941] Container 'jupyter-user-5fxyp2tko8xi' is gone
[I 2021-03-29 14:49:20.779 JupyterHub dockerspawner:1162] Created container jupyter-user-5fxyp2tko8xi (id: ccc98e0) from image exam-scipy-notebook
[I 2021-03-29 14:49:20.779 JupyterHub dockerspawner:1186] Starting container jupyter-user-5fxyp2tko8xi (id: ccc98e0)
[I 2021-03-29 14:49:21.679 JupyterHub log:181] 302 GET /hub/spawn -> /hub/spawn-pending/user_xyp2tko8xi (user_xyp2tko8xi@134.28.143.251) 1011.49ms
[I 2021-03-29 14:49:21.720 JupyterHub pages:402] user_xyp2tko8xi is pending spawn
[I 2021-03-29 14:49:21.722 JupyterHub log:181] 200 GET /hub/spawn-pending/user_xyp2tko8xi (user_xyp2tko8xi@134.28.143.251) 4.79ms
[I 2021-03-29 14:49:22.918 JupyterHub log:181] 200 GET /hub/api (@172.20.0.5) 0.79ms
[I 2021-03-29 14:49:22.950 JupyterHub log:181] 200 POST /hub/api/users/user_xyp2tko8xi/activity (user_xyp2tko8xi@172.20.0.5) 22.77ms
[I 2021-03-29 14:49:23.048 JupyterHub base:904] User user_xyp2tko8xi took 2.370 seconds to start
[I 2021-03-29 14:49:23.049 JupyterHub proxy:257] Adding user user_xyp2tko8xi to proxy /user/user_xyp2tko8xi/ => http://172.20.0.5:8888
14:49:23.051 [ConfigProxy] info: Adding route /user/user_xyp2tko8xi -> http://172.20.0.5:8888
14:49:23.051 [ConfigProxy] info: Route added /user/user_xyp2tko8xi -> http://172.20.0.5:8888
14:49:23.052 [ConfigProxy] info: 201 POST /api/routes/user/user_xyp2tko8xi
[I 2021-03-29 14:49:23.054 JupyterHub users:664] Server user_xyp2tko8xi is ready
[I 2021-03-29 14:49:23.054 JupyterHub log:181] 200 GET /hub/api/users/user_xyp2tko8xi/server/progress (user_xyp2tko8xi@134.28.143.251) 1127.92ms
[I 2021-03-29 14:49:23.104 JupyterHub log:181] 302 GET /hub/spawn-pending/user_xyp2tko8xi -> /user/user_xyp2tko8xi/ (user_xyp2tko8xi@134.28.143.251) 4.73ms
[I 2021-03-29 14:49:23.252 JupyterHub log:181] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-user_xyp2tko8xi&redirect_uri=%2Fuser%2Fuser_xyp2tko8xi%2Foauth_callback&response_type=code&state=[secret] -> /user/user_xyp2tko8xi/oauth_callback?code=[secret]&state=[secret] (user_xyp2tko8xi@134.28.143.251) 17.76ms
[I 2021-03-29 14:49:23.332 JupyterHub log:181] 200 POST /hub/api/oauth2/token (user_xyp2tko8xi@172.20.0.5) 24.55ms
[I 2021-03-29 14:49:23.348 JupyterHub log:181] 200 GET /hub/api/authorizations/token/[secret] (user_xyp2tko8xi@172.20.0.5) 11.16ms
14:49:58.393 [ConfigProxy] info: 200 GET /api/routes
[I 2021-03-29 14:49:58.401 JupyterHub proxy:319] Checking routes
[I 2021-03-29 14:50:19.707 JupyterHub log:181] 200 POST /hub/api/users/admin/activity (admin@172.20.0.4) 11.28ms
[I 2021-03-29 14:52:49.117 JupyterHub log:181] 200 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-admin&redirect_uri=%2Fuser%2Fadmin%2Foauth_callback&response_type=code&state=[secret] (user_xyp2tko8xi@134.28.143.251) 21.48ms
[I 2021-03-29 14:52:49.158 JupyterHub log:181] 200 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-admin&redirect_uri=%2Fuser%2Fadmin%2Foauth_callback&response_type=code&state=[secret] (user_xyp2tko8xi@134.28.143.251) 5.77ms
[I 2021-03-29 14:52:49.648 JupyterHub log:181] 200 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-admin&redirect_uri=%2Fuser%2Fadmin%2Foauth_callback&response_type=code&state=[secret] (user_xyp2tko8xi@134.28.143.251) 6.45ms
[I 2021-03-29 14:52:49.694 JupyterHub log:181] 200 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-admin&redirect_uri=%2Fuser%2Fadmin%2Foauth_callback&response_type=code&state=[secret] (user_xyp2tko8xi@134.28.143.251) 11.64ms
...
The JupyterHub Docker container logs
[D 2021-03-29 14:45:18.683 SingleUserNotebookApp auth:874] Allowing Hub user admin
[D 2021-03-29 14:45:18.684 SingleUserNotebookApp log:181] 304 GET /user/admin/custom/custom.js?v=20210329144509 (admin@134.28.143.251) 1.68ms
[D 2021-03-29 14:45:18.729 SingleUserNotebookApp auth:874] Allowing Hub user admin
[I 2021-03-29 14:45:18.730 SingleUserNotebookApp log:181] 101 GET /user/admin/terminals/websocket/1 (admin@134.28.143.251) 1.58ms
[D 2021-03-29 14:45:18.731 SingleUserNotebookApp zmqhandlers:154] Opening websocket /user/admin/terminals/websocket/1
TermSocket.open: 1
TermSocket.open: Opened 1
Websocket closed
[D 2021-03-29 14:49:00.515 SingleUserNotebookApp auth:874] Allowing Hub user admin
[I 2021-03-29 14:49:00.517 SingleUserNotebookApp log:181] 302 GET /user/admin/logout -> /hub/logout (admin@134.28.143.251) 2.96ms
[D 2021-03-29 14:50:19.691 SingleUserNotebookApp mixins:499] Notifying Hub of activity 2021-03-29T14:48:58.680381Z
[D 2021-03-29 14:52:49.018 SingleUserNotebookApp auth:504] No user identified
[D 2021-03-29 14:52:49.021 SingleUserNotebookApp auth:835] Redirecting to login url: /hub/api/oauth2/authorize?client_id=jupyterhub-user-admin&redirect_uri=%2Fuser%2Fadmin%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogImQwMzkxYjU2NTE0ZDRiYmI4ZWY2MmE1ZTNmODNiOTM4IiwgIm5leHRfdXJsIjogIi91c2VyL2FkbWluL2FwaS9zZXNzaW9ucz9fPTE2MTcwMjkxMTIzNzUifQ
[I 2021-03-29 14:52:49.022 SingleUserNotebookApp log:181] 302 GET /user/admin/api/sessions?_=1617029112375 -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-admin&redirect_uri=%2Fuser%2Fadmin%2Foauth_callback&response_type=code&state=[secret] (@134.28.143.251) 4.82ms
[D 2021-03-29 14:52:49.057 SingleUserNotebookApp auth:504] No user identified
[D 2021-03-29 14:52:49.058 SingleUserNotebookApp auth:835] Redirecting to login url: /hub/api/oauth2/authorize?client_id=jupyterhub-user-admin&redirect_uri=%2Fuser%2Fadmin%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogImFiY2JjYzliZWM2NzQ0NWE4ZDk4OTQ5NThiZTExYjk0IiwgIm5leHRfdXJsIjogIi91c2VyL2FkbWluL2FwaS90ZXJtaW5hbHM_Xz0xNjE3MDI5MTEyMzc2In0
[I 2021-03-29 14:52:49.059 SingleUserNotebookApp log:181] 302 GET /user/admin/api/terminals?_=1617029112376 -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-admin&redirect_uri=%2Fuser%2Fadmin%2Foauth_callback&response_type=code&state=[secret] (@134.28.143.251) 2.23ms
[D 2021-03-29 14:52:49.570 SingleUserNotebookApp auth:504] No user identified
[W 2021-03-29 14:52:49.570 SingleUserNotebookApp auth:682] Detected unused OAuth state cookies
[D 2021-03-29 14:52:49.571 SingleUserNotebookApp auth:835] Redirecting to login url: /hub/api/oauth2/authorize?client_id=jupyterhub-user-admin&redirect_uri=%2Fuser%2Fadmin%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogImMwMzNhOGFlYmUwODQxOTA5ODgyODQ5NzE5MDJlZjlmIiwgIm5leHRfdXJsIjogIi91c2VyL2FkbWluL2FwaS9zZXNzaW9ucz9fPTE2MTcwMjgyNDIyMTMiLCAiY29va2llX25hbWUiOiAianVweXRlcmh1Yi11c2VyLWFkbWluLW9hdXRoLXN0YXRlLXRyQktERFFzIn0
[I 2021-03-29 14:52:49.571 SingleUserNotebookApp log:181] 302 GET /user/admin/api/sessions?_=1617028242213 -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-admin&redirect_uri=%2Fuser%2Fadmin%2Foauth_callback&response_type=code&state=[secret] (@134.28.143.251) 1.90ms
[D 2021-03-29 14:52:49.606 SingleUserNotebookApp auth:504] No user identified
[W 2021-03-29 14:52:49.607 SingleUserNotebookApp auth:682] Detected unused OAuth state cookies
[D 2021-03-29 14:52:49.607 SingleUserNotebookApp auth:835] Redirecting to login url: /hub/api/oauth2/authorize?client_id=jupyterhub-user-admin&redirect_uri=%2Fuser%2Fadmin%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogImExYzYwNzhmNjAxZjQzYmFhNTdhMDJmY2YxYzUwNDQ3IiwgIm5leHRfdXJsIjogIi91c2VyL2FkbWluL2FwaS90ZXJtaW5hbHM_Xz0xNjE3MDI4MjQyMjE0IiwgImNvb2tpZV9uYW1lIjogImp1cHl0ZXJodWItdXNlci1hZG1pbi1vYXV0aC1zdGF0ZS1EUndRWXRwbiJ9
[I 2021-03-29 14:52:49.608 SingleUserNotebookApp log:181] 302 GET /user/admin/api/terminals?_=1617028242214 -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-admin&redirect_uri=%2Fuser%2Fadmin%2Foauth_callback&response_type=code&state=[secret] (@134.28.143.251) 2.22ms
[D 2021-03-29 14:54:56.186 SingleUserNotebookApp mixins:499] Notifying Hub of activity 2021-03-29T14:48:58.680381Z
[D 2021-03-29 14:57:52.023 SingleUserNotebookApp auth:504] No user identified
[W 2021-03-29 14:57:52.023 SingleUserNotebookApp auth:682] Detected unused OAuth state cookies
[D 2021-03-29 14:57:52.024 SingleUserNotebookApp auth:835] Redirecting to login url: /hub/api/oauth2/authorize?client_id=jupyterhub-user-admin&redirect_uri=%2Fuser%2Fadmin%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogIjRkMTI4ZTY3MmFjYzQzY2I5N2Q3NzhmMzI1NGI5YjYyIiwgIm5leHRfdXJsIjogIi91c2VyL2FkbWluL2FwaS9zZXNzaW9ucz9fPTE2MTcwMjgyNDIyMTUiLCAiY29va2llX25hbWUiOiAianVweXRlcmh1Yi11c2VyLWFkbWluLW9hdXRoLXN0YXRlLUtydVhBRWZkIn0
Issue Analytics
- State:
- Created 2 years ago
- Comments:7 (2 by maintainers)
Top Results From Across the Web
Attach Handlers to Container Lifecycle Events - Kubernetes
This page shows how to attach handlers to Container lifecycle events. Kubernetes supports the postStart and preStop events.
Read more >debugging a postStart hook - Google Groups
I can't find errors produced in this script in the log. I'm forced to redirect stdout and stderr on every cmd in this...
Read more >Debugging Kubernetes postStart Hooks - Breaking Computer
I recently ran into an issue where running a postStart hook in a Kubernetes Pod was failing. Kubernetes is normally pretty good about...
Read more >containerd-shim process isn't reaped for some killed containers
Docker thinks the container is running, although it is not; Containerd shows the task is stopped; The container-shim process wasn't reaped and ...
Read more >Kubernetes Container Lifecycle Events and Hooks
PostStart : This hook is executed right after a container is created. However, the hook might be invoked after the container's ENTRYPOINT is ......
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Thank you very much, @manics ! Your hint with
sh -c '...'solved it for me!However, here are the logs that I would expect to contain some indication:
and
Now you might guess that in one cell of a Jupyter Notebook I added a
!wget https://google.com, saved it and restarted my Jupyter Notebook server. But you can’t read it from the logs. However, my expections would be thatc.DockerSpawner.debug = Truewould provide me with exactly that information.I am not sure how much should be logged from the Docker client in general. If in my case all stdout and stderr had been dumped to
DEBUG, that would have been a great help! SoDEBUGwould be my first suggestion. If one decides to do the logging at the Docker client, I guess it should be placed somewhere close to https://github.com/jupyterhub/dockerspawner/blob/master/dockerspawner/dockerspawner.py#L903. But most of the times things work fine and it could create a lot of noise. Therefore, maybe a use-case specific handling at https://github.com/jupyterhub/dockerspawner/blob/master/dockerspawner/dockerspawner.py#L745 could be more straight-forward.Thanks, for me the pull request #427 solved this perfectly!