kubespawner: pending spawn in loop until urllib3.exceptions.ReadTimeoutError generated

I use jupyter with kubespawner (k8s in azure), when jupyterhub run more than 2 hours, it doesn’t create new pod for logged user. Log:

[I 2018-11-29 08:25:24.998 JupyterHub log:158] 302 GET /user/toupinets/ -> /hub/user/toupinets/ (@10.1.32.66) 0.74ms
[I 2018-11-29 08:25:28.250 JupyterHub base:1012] Pending spawn for toupinets didn't finish in 3.0 seconds
[I 2018-11-29 08:25:28.250 JupyterHub base:1018] toupinets is pending spawn

…

[W 2018-11-29 08:53:27.810 JupyterHub user:471] toupinets's server failed to start in 600 seconds, giving up
[E 2018-11-29 08:53:28.258 JupyterHub gen:974] Exception in Future <Task finished coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py:619> exception=TimeoutError('Timeout',)> after timeout

… after some time it generate Exception:

[E 2018-11-29 08:59:07.068 JupyterHub gen:974] Exception in Future <Future finished exception=ReadTimeoutError("HTTPSConnectionPool(host='main-370a339f.hcp.southeastasia.azmk8s.io', port=443): Read timed out. (read timeout=None)",)> after timeout

It start work… It is possible to define read timeout for kubernetes service?

[andreitoupinets]

@yvan , because azure load balancer drop connection after ~5min inactivity. Try keepalive, just put into jupyterhub_config.py

import socket
from urllib3 import connection

connection.HTTPConnection.default_socket_options += [(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1),
                                                    (socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, 60),
                                                    (socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, 60),
                                                    (socket.IPPROTO_TCP, socket.TCP_KEEPCNT, 3)]

[meeseeksmachine]

This issue has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/spawn-failed-timeout-even-when-start-timeout-is-set-to-3600-seconds/8098/2