kubespawner stopped working after kubernetes upgrade from 1.15 to 1.16

[E 2019-09-25 05:16:53.998 JupyterHub web:1788] Uncaught exception GET /hub/user/apoliakevitch/ (10.244.4.1)
    HTTPServerRequest(protocol='http', host='conda.corp-apps.com', method='GET', uri='/hub/user/apoliakevitch/', version='HTTP/1.1', remote_ip='10.244.4.1')
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/tornado/web.py", line 1699, in _execute
        result = await result
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py", line 1013, in get
        raise copy.copy(exc).with_traceback(exc.__traceback__)
      File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 589, in error_callback
        future.result()
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py", line 636, in finish_user_spawn
        await spawn_future
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/user.py", line 489, in spawn
        raise e
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/user.py", line 409, in spawn
        url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
      File "/usr/local/lib/python3.6/dist-packages/kubespawner/spawner.py", line 1636, in _start
        events = self.events
      File "/usr/local/lib/python3.6/dist-packages/kubespawner/spawner.py", line 1491, in events
        for event in self.event_reflector.events:
      File "/usr/local/lib/python3.6/dist-packages/kubespawner/spawner.py", line 72, in events
        key=lambda x: x.last_timestamp,
    TypeError: '<' not supported between instances of 'datetime.datetime' and 'NoneType'

[michitaro]

The patch can be done also by the command below.

kubectl patch deploy -n $NAMESPACE hub --type json --patch '[{"op": "replace", "path": "/spec/template/spec/containers/0/command", "value": ["bash", "-c", "\nmkdir -p ~/hotfix\ncp -r /usr/local/lib/python3.6/dist-packages/kubespawner ~/hotfix\nls -R ~/hotfix\npatch ~/hotfix/kubespawner/spawner.py << EOT\n72c72\n<             key=lambda x: x.last_timestamp,\n---\n>             key=lambda x: x.last_timestamp and x.last_timestamp.timestamp() or 0.,\nEOT\n\nPYTHONPATH=$HOME/hotfix jupyterhub --config /srv/jupyterhub_config.py --upgrade-db\n"]}]'

[michitaro]

@dmpe Change deployment hub’s .spec.template.spec.containers[0].command to something like this:

kubectl edit deploy -n $NAMESPACE hub

...
      containers:
      - command:
        - bash
        - -c
        - |
          mkdir -p ~/hotfix
          cp -r /usr/local/lib/python3.6/dist-packages/kubespawner ~/hotfix
          ls -R ~/hotfix
          patch ~/hotfix/kubespawner/spawner.py << EOT
          72c72
          <             key=lambda x: x.last_timestamp,
          ---
          >             key=lambda x: x.last_timestamp and x.last_timestamp.timestamp() or 0.,
          EOT

          PYTHONPATH=$HOME/hotfix jupyterhub --config /srv/jupyterhub_config.py --upgrade-db
        env:
        - name: PYTHONUNBUFFERED
          value: "1"
        - name: HELM_RELEASE_NAME
...