jupyterhub pod fails to init after upgrade

Bug description

hub pod fails to initialize after upgrade

How to reproduce

sudo helm upgrade binderhub jupyterhub/binderhub --version=v0.2.0-n852.h7c39292 -f config.yaml -f secrets.yaml -n binderhub

Your personal set up

• binderhub: 0.2.0-n845.hcc57b24
• k8s v1.19.3
• OS: ubuntu 18

paste relevant logs here, if any

NAME                                             READY   STATUS             RESTARTS   AGE	NODE                     START
binder-68ff98c78d-svr4z                          1/1     Running            0          2m28s	neurolibre-test-node1    2021-11-22T17:24:23Z
binderhub-image-cleaner-5srbp                    1/1     Running            0          26d	neurolibre-test-node1    2021-10-27T15:14:53Z
binderhub-image-cleaner-t9h56                    1/1     Running            0          26d	neurolibre-test-master   2021-10-27T15:14:53Z
binderhub-proxy-ingress-nginx-controller-8k6jf   1/1     Running            0          26d	neurolibre-test-master   2021-10-26T21:25:30Z
binderhub-proxy-ingress-nginx-controller-q5cjr   1/1     Running            0          26d	neurolibre-test-node1    2021-10-26T21:25:30Z
continuous-image-puller-tgvvt                    1/1     Running            0          12d	neurolibre-test-master   2021-11-10T16:47:25Z
continuous-image-puller-w5hjp                    1/1     Running            0          12d	neurolibre-test-node1    2021-11-10T16:47:24Z
hub-5d77dd5cf7-qjbpr                             0/1     CrashLoopBackOff   4          2m17s	neurolibre-test-master   2021-11-22T17:24:34Z

Loading /usr/local/etc/jupyterhub/secret/values.yaml
No config at /usr/local/etc/jupyterhub/existing-secret/values.yaml
Loading extra config: 0-binderspawnermixin
Loading extra config: 00-binder
[I 2021-11-22 17:28:08.272 JupyterHub app:2459] Running JupyterHub version 1.4.2
[I 2021-11-22 17:28:08.272 JupyterHub app:2489] Using Authenticator: nullauthenticator.NullAuthenticator-1.0.0
[I 2021-11-22 17:28:08.272 JupyterHub app:2489] Using Spawner: builtins.BinderSpawner
[I 2021-11-22 17:28:08.272 JupyterHub app:2489] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.4.2
[I 2021-11-22 17:28:08.286 JupyterHub dbutil:130] Upgrading sqlite:///jupyterhub.sqlite
[I 2021-11-22 17:28:08.286 JupyterHub dbutil:99] Backing up jupyterhub.sqlite => jupyterhub.sqlite.2021-11-22-172808
[I 2021-11-22 17:28:08.680 alembic.runtime.migration migration:164] Context impl SQLiteImpl.
[I 2021-11-22 17:28:08.680 alembic.runtime.migration migration:167] Will assume non-transactional DDL.
[E 2021-11-22 17:28:08.684 alembic.util.messaging messaging:60] Can't locate revision identified by '833da8570507'
FAILED: Can't locate revision identified by '833da8570507'
[E 2021-11-22 17:28:08.738 JupyterHub app:2969]
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/app.py", line 2966, in launch_instance_async
        await self.initialize(argv)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/app.py", line 2501, in initialize
        self.init_db()
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/app.py", line 1703, in init_db
        dbutil.upgrade_if_needed(self.db_url, log=self.log)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/dbutil.py", line 135, in upgrade_if_needed
        upgrade(db_url)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/dbutil.py", line 84, in upgrade
        check_call(['alembic', '-c', alembic_ini, 'upgrade', revision])
      File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['alembic', '-c', '/tmp/tmpyfkmw4bu/alembic.ini', 'upgrade', 'head']' returned non-zero exit status 255.

[minrk]

if you have some instructions on how I can revert to “old” (<2.0) jupyter config let me know.

That would be copying the backup file created when you first upgraded to 2.0, as I tried to describe:

“restoring” means copying a file from jupyterhub.sqlite.YYYY-MM-DD-TTTTTT to jupyterhub.sqlite in the hub-db volume.

If you are using this for binderhub, however, nothing in the database is usually valid for more than a day, so just deleting or renaming jupyterhub.sqlite and then restarting the hub pod should probably do it. Doing that would orphan any running pods, but so would restoring the database from an older state.

[ltetrel]

In my case I have a persistent volume k8s request for the jupyterhub database. So I can modify files on my host:

kind: PersistentVolume
apiVersion: v1
metadata:
  name: hub-db-dir
  labels:
    type: local
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/shared"