Upgrade to a newer Python version documentation not functional for python3.10.x

Bug description

Following the instructions in the documentation on how to Upgrade to a newer Python version, no users are able to create a new server, admin or not.

The default TLJH setup works fine, and I’ve been able to run it as is on a manual setup w/HTTPs+etc. just fine. However, attempting to upgrade the python version for users to 3.10.x, the server functionality is entirely broken.

Expected behaviour

Following the steps in the documentation should result in the desired state (upgraded python version) with a still functional hub setup.

Actual behaviour

The hub fails to create any new user server’s after the doc steps have been following (slightly modified so they actually work, bc as they’re written now are also broken; the ... @ file:///... conda packages all get removed and are thus broken when the last pip install ... is attempted. I’ve tried simply removing those packages entirely, just removing the @ ... and also doing pip list --format=freeze and using that package, none of which seem to have any effect on the problem here, which is the hub being unable to create any new user servers after upgrade.

journalctl logs (see below logs section) for the user’s process seems to show it complaining about a jupyterhub package missing, but if I manually check the script that it say’s is being invoked, it seems to show that is installed, and that the python at the script’s shebang can even import it itself?:

cat /opt/tljh/hub/bin/jupyterhub-singleuser
#!/opt/tljh/hub/bin/python
...

# executing interpreter using above /opt/tljh/hub/bin/python
Python 3.10.4 (main, Jun 29 2022, 12:14:53) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import jupyterhub
>>>

and

/opt/tljh/user/bin/python -m pip list | grep jupy
jupyter-client         7.3.4
jupyter-core           4.11.1
jupyter-resource-usage 0.6.1
jupyter-server         1.18.1
jupyter-telemetry      0.1.0
jupyterhub             1.5.0
jupyterlab             3.4.3
jupyterlab-pygments    0.2.2
jupyterlab-server      2.15.0
jupyterlab-widgets     1.1.1
nteract-on-jupyter     2.1.3

How to reproduce

1. Install TLJH on a fresh server installation
2. Manually access the server running TLJH, and follow the steps to upgrade to a newer python version:

# been doing this as a non root user? the docs don't indicate either way; presumably is fine since latter steps use sudo?
source /opt/tljh/user/bin/activate

pip freeze > pip_pkgs.txt # this needs to be fixed/changed too, but isn't this problem -- also, this shouldn't be root?

sudo env PATH=${PATH} conda update --all # this also needed to be fixed, should be `sudo env PATH=...` otherwise it still can't find conda?

sudo env PATH=${PATH} conda install python=3.10 # same env problem as above

pip install -r pip_pkgs.txt # this will fail unless you've removed the conda packages/used the pip list --format=freeze command to generate
# this pip install seems to put everything into the user's `~/.local/bin` and complains about PATH?

1. Access the Hub, and sign in; first time admin login for example
2. Go to hub homepage or click the button to create your user’s server
3. Wait for create to fail and timeout (this fails for the first user ie initial admin login, but I also created additional non-admin users and tested their server’s too and they failed to start for the same reason)

Your personal set up

• OS:
  • Distributor ID: Ubuntu
  • Description: Ubuntu 22.04 LTS
  • Release: 22.04
  • Codename: jammy

• Version(s):
• Before upgrade:

  • conda --version
    conda 4.10.3
    
    [/opt/tljh/hub/bin/python, /opt/tljh/hub/bin/python3, /opt/tljh/hub/bin/python3.10] --version
    Python 3.10.4
    
    [/opt/tljh/user/bin/python, /opt/tljh/user/bin/python3, /opt/tljh/user/bin/python3.9] --version
    Python 3.9.7
    
    /opt/tljh/hub/bin/jupyterhub --version 
    1.5.0
    
    /opt/tljh/user/condabin/conda --version
    conda 4.10.3
    
    /opt/tljh/user/bin/jupyterhub --version
    1.5.0
    

• After upgrade:

  • conda --version
    conda 4.13.0
    
    [/opt/tljh/hub/bin/python, /opt/tljh/hub/bin/python3, /opt/tljh/hub/bin/python3.10] --version
    Python 3.10.4
    
    [/opt/tljh/user/bin/python, /opt/tljh/user/bin/python3, /opt/tljh/user/bin/python3.10] --version
    Python 3.10.5
    
    /opt/tljh/hub/bin/jupyterhub --version 
    1.5.0
    
    /opt/tljh/user/condabin/conda --version
    conda 4.13.0
    
    /opt/tljh/user/bin/jupyterhub --version
    1.5.0
    

Full environment

# from inside the conda venv, BEFORE upgrade
alembic==1.8.1
anyio==3.6.1
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
asttokens==2.0.5
async-generator==1.10
attrs==21.4.0
Babel==2.10.3
backcall==0.2.0
beautifulsoup4==4.11.1
bleach==5.0.1
brotlipy==0.7.0
certifi==2022.6.15
certipy==0.1.3
cffi==1.14.6
chardet==4.0.0
charset-normalizer==2.0.0
colorama==0.4.4
conda==4.10.3
conda-package-handling==1.7.3
cryptography==3.4.8
debugpy==1.6.2
decorator==5.1.1
defusedxml==0.7.1
entrypoints==0.4
executing==0.8.3
fastjsonschema==2.16.1
greenlet==1.1.2
idna==3.1
importlib-metadata==4.12.0
ipykernel==6.15.1
ipython==8.4.0
ipython-genutils==0.2.0
ipywidgets==7.7.1
jedi==0.18.1
Jinja2==3.1.2
json5==0.9.8
jsonschema==4.7.2
jupyter-client==7.3.4
jupyter-core==4.11.1
jupyter-resource-usage==0.6.1
jupyter-server==1.18.1
jupyter-telemetry==0.1.0
jupyterhub==1.5.0
jupyterlab==3.4.3
jupyterlab-pygments==0.2.2
jupyterlab-server==2.15.0
jupyterlab-widgets==1.1.1
Mako==1.2.1
mamba==0.16.0
MarkupSafe==2.1.1
matplotlib-inline==0.1.3
mistune==0.8.4
nbclassic==0.4.3
nbclient==0.6.6
nbconvert==6.5.0
nbformat==5.4.0
nbgitpuller==1.1.0
nest-asyncio==1.5.5
notebook==6.4.12
notebook-shim==0.1.0
nteract-on-jupyter==2.1.3
oauthlib==3.2.0
packaging==21.3
pamela==1.0.0
pandocfilters==1.5.0
parso==0.8.3
pexpect==4.8.0
pickleshare==0.7.5
pip==21.3
prometheus-client==0.14.1
prompt-toolkit==3.0.30
psutil==5.9.1
ptyprocess==0.7.0
pure-eval==0.2.2
pycosat==0.6.3
pycparser==2.20
Pygments==2.12.0
pyOpenSSL==21.0.0
pyparsing==3.0.9
pyrsistent==0.18.1
PySocks==1.7.1
python-dateutil==2.8.2
python-json-logger==2.0.4
pytz==2022.1
pyzmq==23.2.0
requests==2.26.0
ruamel.yaml==0.17.21
ruamel.yaml.clib==0.2.6
ruamel-yaml-conda==0.15.80
Send2Trash==1.8.0
setuptools==58.2.0
six==1.16.0
sniffio==1.2.0
soupsieve==2.3.2.post1
SQLAlchemy==1.4.39
stack-data==0.3.0
terminado==0.15.0
tinycss2==1.1.1
tornado==6.2
tqdm==4.62.3
traitlets==5.3.0
urllib3==1.26.7
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==1.3.3
wheel==0.37.0
widgetsnbextension==3.6.1
zipp==3.8.1

from inside venv, AFTER upgrade

alembic==1.8.1
anyio==3.6.1
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
asttokens==2.0.5
async-generator==1.10
attrs==21.4.0
Babel==2.10.3
backcall==0.2.0
beautifulsoup4==4.11.1
bleach==5.0.1
brotlipy==0.7.0
certifi==2022.6.15
certipy==0.1.3
cffi==1.15.1
chardet==4.0.0
charset-normalizer==2.1.0
colorama==0.4.5
conda==4.13.0
conda-package-handling==1.8.1
cryptography==37.0.4
debugpy==1.6.2
decorator==5.1.1
defusedxml==0.7.1
entrypoints==0.4
executing==0.8.3
fastjsonschema==2.16.1
greenlet==1.1.2
idna==3.3
importlib-metadata==4.12.0
ipykernel==6.15.1
ipython==8.4.0
ipython-genutils==0.2.0
ipywidgets==7.7.1
jedi==0.18.1
Jinja2==3.1.2
json5==0.9.8
jsonschema==4.7.2
jupyter-client==7.3.4
jupyter-core==4.11.1
jupyter-resource-usage==0.6.1
jupyter-server==1.18.1
jupyter-telemetry==0.1.0
jupyterhub==1.5.0
jupyterlab==3.4.3
jupyterlab-pygments==0.2.2
jupyterlab-server==2.15.0
jupyterlab-widgets==1.1.1
libmambapy==0.24.0
Mako==1.2.1
mamba==0.24.0
MarkupSafe==2.1.1
matplotlib-inline==0.1.3
mistune==0.8.4
nbclassic==0.4.3
nbclient==0.6.6
nbconvert==6.5.0
nbformat==5.4.0
nbgitpuller==1.1.0
nest-asyncio==1.5.5
notebook==6.4.12
notebook-shim==0.1.0
nteract-on-jupyter==2.1.3
oauthlib==3.2.0
packaging==21.3
pamela==1.0.0
pandocfilters==1.5.0
parso==0.8.3
pexpect==4.8.0
pickleshare==0.7.5
pip==21.3
prometheus-client==0.14.1
prompt-toolkit==3.0.30
psutil==5.9.1
ptyprocess==0.7.0
pure-eval==0.2.2
pycosat==0.6.3
pycparser==2.21
Pygments==2.12.0
pyOpenSSL==22.0.0
pyparsing==3.0.9
pyrsistent==0.18.1
PySocks==1.7.1
python-dateutil==2.8.2
python-json-logger==2.0.4
pytz==2022.1
pyzmq==23.2.0
requests==2.28.1
ruamel.yaml==0.17.21
ruamel.yaml.clib==0.2.6
ruamel-yaml-conda==0.15.80
Send2Trash==1.8.0
setuptools==58.2.0
six==1.16.0
sniffio==1.2.0
soupsieve==2.3.2.post1
SQLAlchemy==1.4.39
stack-data==0.3.0
terminado==0.15.0
tinycss2==1.1.1
tornado==6.2
tqdm==4.64.0
traitlets==5.3.0
urllib3==1.26.10
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==1.3.3
wheel==0.37.0
widgetsnbextension==3.6.1
zipp==3.8.1

Configuration

# from inside the venv, BEFORE upgrade:
grep -v '\(^#\|^[[:space:]]*$\)' jupyterhub_config.py
<nothing returned; default conf>

Logs

For journalctl -u jupyterhub when trying to start a user’s server AFTER upgrade:

Jul 18 22:32:51 test-tljh-server python3[6577]: [I 2022-07-18 22:32:51.883 JupyterHub log:189] 302 GET / -> /hub/ (@127.0.0.1) 1.13ms
Jul 18 22:32:52 test-tljh-server python3[6577]: [W 2022-07-18 22:32:52.024 JupyterHub base:390] Invalid or expired cookie token
Jul 18 22:32:52 test-tljh-server python3[6577]: [I 2022-07-18 22:32:52.025 JupyterHub log:189] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@127.0.0.1) 1.51ms
Jul 18 22:32:52 test-tljh-server python3[6577]: [I 2022-07-18 22:32:52.115 JupyterHub log:189] 200 GET /hub/login?next=%2Fhub%2F (@127.0.0.1) 24.61ms
Jul 18 22:32:58 test-tljh-server python3[6577]: [I 2022-07-18 22:32:58.718 JupyterHub base:762] User logged in: testoperator
Jul 18 22:32:58 test-tljh-server python3[6577]: [I 2022-07-18 22:32:58.718 JupyterHub log:189] 302 POST /hub/login?next=%2Fhub%2F -> /hub/ (testoperator@127.0.0.1) 246.62ms
Jul 18 22:32:58 test-tljh-server python3[6577]: [I 2022-07-18 22:32:58.909 JupyterHub log:189] 302 GET /hub/ -> /hub/spawn (testoperator@127.0.0.1) 42.63ms
Jul 18 22:32:59 test-tljh-server python3[6577]: [I 2022-07-18 22:32:59.001 JupyterHub provider:574] Creating oauth client jupyterhub-user-testoperator
Jul 18 22:32:59 test-tljh-server useradd[8684]: new group: name=jupyter-testoperator, GID=1004
Jul 18 22:32:59 test-tljh-server useradd[8684]: new user: name=jupyter-testoperator, UID=1002, GID=1004, home=/home/jupyter-testoperator, shell=/bin/sh, from=none
Jul 18 22:32:59 test-tljh-server python3[8693]: Adding user jupyter-testoperator to group jupyterhub-users
Jul 18 22:32:59 test-tljh-server gpasswd[8693]: user jupyter-testoperator added by root to group jupyterhub-users
Jul 18 22:32:59 test-tljh-server python3[8699]: Adding user jupyter-testoperator to group jupyterhub-admins
Jul 18 22:32:59 test-tljh-server gpasswd[8699]: user jupyter-testoperator added by root to group jupyterhub-admins
Jul 18 22:32:59 test-tljh-server python3[8709]: Running as unit: jupyter-testoperator.service
Jul 18 22:32:59 test-tljh-server python3[6577]: [I 2022-07-18 22:32:59.991 JupyterHub log:189] 302 GET /hub/spawn -> /hub/spawn-pending/testoperator (testoperator@127.0.0.1) 1005.75ms
Jul 18 22:33:00 test-tljh-server python3[6577]: [I 2022-07-18 22:33:00.098 JupyterHub pages:402] testoperator is pending spawn
Jul 18 22:33:00 test-tljh-server python3[6577]: [I 2022-07-18 22:33:00.102 JupyterHub log:189] 200 GET /hub/spawn-pending/testoperator (testoperator@127.0.0.1) 5.87ms
Jul 18 22:33:08 test-tljh-server python3[6577]: Task exception was never retrieved
Jul 18 22:33:08 test-tljh-server python3[6577]: future: <Task finished name='Task-181' coro=<BaseHandler.spawn_single_user() done, defined at /opt/tljh/hub/lib/python3.10/site-packages/jupyterhub/handlers/base.py:796> exception=HTTPError()>
Jul 18 22:33:08 test-tljh-server python3[6577]: Traceback (most recent call last):
Jul 18 22:33:08 test-tljh-server python3[6577]:   File "/opt/tljh/hub/lib/python3.10/site-packages/jupyterhub/handlers/base.py", line 996, in spawn_single_user
Jul 18 22:33:08 test-tljh-server python3[6577]:     await gen.with_timeout(
Jul 18 22:33:08 test-tljh-server python3[6577]: asyncio.exceptions.TimeoutError: Timeout
Jul 18 22:33:08 test-tljh-server python3[6577]: During handling of the above exception, another exception occurred:
Jul 18 22:33:08 test-tljh-server python3[6577]: Traceback (most recent call last):
Jul 18 22:33:08 test-tljh-server python3[6577]:   File "/opt/tljh/hub/lib/python3.10/site-packages/jupyterhub/handlers/base.py", line 1030, in spawn_single_user
Jul 18 22:33:08 test-tljh-server python3[6577]:     raise web.HTTPError(
Jul 18 22:33:08 test-tljh-server python3[6577]: tornado.web.HTTPError: HTTP 500: Internal Server Error (Spawner failed to start [status=1]. The logs for testoperator may contain details.)
Jul 18 22:33:19 test-tljh-server python3[6577]: [I 2022-07-18 22:33:19.178 JupyterHub log:189] 200 GET /hub/api/ (cull-idle@127.0.0.1) 8.13ms
Jul 18 22:33:19 test-tljh-server python3[6577]: [I 2022-07-18 22:33:19.182 JupyterHub log:189] 200 GET /hub/api/users?state=[secret] (cull-idle@127.0.0.1) 2.71ms
Jul 18 22:33:31 test-tljh-server python3[6577]: [W 2022-07-18 22:33:31.817 JupyterHub user:767] testoperator's server never showed up at http://127.0.0.1:46559/user/testoperator/ after 30 seconds. Giving up
Jul 18 22:33:31 test-tljh-server python3[6577]: [E 2022-07-18 22:33:31.844 JupyterHub gen:630] Exception in Future <Task finished name='Task-182' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /opt/tljh/hub/lib/python3.10/site-packages/jupyterhub/handlers/base.py:900> exception=TimeoutError("Server at http://127.0.0.1:46559/user/testoperator/ didn't respond in 30 seconds")> after timeout
Jul 18 22:33:31 test-tljh-server python3[6577]:     Traceback (most recent call last):
Jul 18 22:33:31 test-tljh-server python3[6577]:       File "/opt/tljh/hub/lib/python3.10/site-packages/tornado/gen.py", line 625, in error_callback
Jul 18 22:33:31 test-tljh-server python3[6577]:         future.result()
Jul 18 22:33:31 test-tljh-server python3[6577]:       File "/opt/tljh/hub/lib/python3.10/site-packages/jupyterhub/handlers/base.py", line 907, in finish_user_spawn
Jul 18 22:33:31 test-tljh-server python3[6577]:         await spawn_future
Jul 18 22:33:31 test-tljh-server python3[6577]:       File "/opt/tljh/hub/lib/python3.10/site-packages/jupyterhub/user.py", line 748, in spawn
Jul 18 22:33:31 test-tljh-server python3[6577]:         await self._wait_up(spawner)
Jul 18 22:33:31 test-tljh-server python3[6577]:       File "/opt/tljh/hub/lib/python3.10/site-packages/jupyterhub/user.py", line 795, in _wait_up
Jul 18 22:33:31 test-tljh-server python3[6577]:         raise e
Jul 18 22:33:31 test-tljh-server python3[6577]:       File "/opt/tljh/hub/lib/python3.10/site-packages/jupyterhub/user.py", line 762, in _wait_up
Jul 18 22:33:31 test-tljh-server python3[6577]:         resp = await server.wait_up(
Jul 18 22:33:31 test-tljh-server python3[6577]:       File "/opt/tljh/hub/lib/python3.10/site-packages/jupyterhub/utils.py", line 236, in wait_for_http_server
Jul 18 22:33:31 test-tljh-server python3[6577]:         re = await exponential_backoff(
Jul 18 22:33:31 test-tljh-server python3[6577]:       File "/opt/tljh/hub/lib/python3.10/site-packages/jupyterhub/utils.py", line 184, in exponential_backoff
Jul 18 22:33:31 test-tljh-server python3[6577]:         raise TimeoutError(fail_message)
Jul 18 22:33:31 test-tljh-server python3[6577]:     TimeoutError: Server at http://127.0.0.1:46559/user/testoperator/ didn't respond in 30 seconds
Jul 18 22:33:31 test-tljh-server python3[6577]:
Jul 18 22:33:31 test-tljh-server python3[6577]: [I 2022-07-18 22:33:31.846 JupyterHub log:189] 200 GET /hub/api/users/testoperator/server/progress (testoperator@127.0.0.1) 31593.27ms

For journalctl -u testoperator.service:

Jul 18 22:32:59 test-tljh-server systemd[1]: Started /bin/bash -c cd /home/jupyter-testoperator && exec jupyterhub-singleuser --port=46559.
Jul 18 22:32:59 test-tljh-server bash[8711]: Traceback (most recent call last):
Jul 18 22:32:59 test-tljh-server bash[8711]:   File "/opt/tljh/user/bin/jupyterhub-singleuser", line 5, in <module>
Jul 18 22:32:59 test-tljh-server bash[8711]:     from jupyterhub.singleuser import main
Jul 18 22:32:59 test-tljh-server bash[8711]: ModuleNotFoundError: No module named 'jupyterhub'
Jul 18 22:32:59 test-tljh-server systemd[1]: jupyter-testoperator.service: Main process exited, code=exited, status=1/FAILURE
Jul 18 22:32:59 test-tljh-server systemd[1]: jupyter-testoperator.service: Failed with result 'exit-code'.

[mtav]

Your solution fails for me at the python 3.10 installation step with the following output:

$ sudo env PATH=${PATH} conda install python=3.10  # has confirmation prompt
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /opt/tljh/user

  added / updated specs:
    - python=3.10


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    brotlipy-0.7.0             |py310h5764c6d_1004         342 KB  conda-forge
    cffi-1.15.1                |  py310h255011f_0         231 KB  conda-forge
    conda-4.14.0               |  py310hff52083_0         1.0 MB  conda-forge
    conda-package-handling-1.8.1|  py310h5764c6d_1        1015 KB  conda-forge
    cryptography-37.0.4        |  py310h597c629_0         1.6 MB  conda-forge
    libmambapy-0.25.0          |  py310hab0e683_2         297 KB  conda-forge
    libnsl-2.0.0               |       h7f98852_0          31 KB  conda-forge
    libsqlite-3.39.3           |       h753d276_0         789 KB  conda-forge
    libuuid-2.32.1             |    h7f98852_1000          28 KB  conda-forge
    mamba-0.25.0               |  py310hf87f941_2          46 KB  conda-forge
    ncurses-6.3                |       h27087fc_1        1002 KB  conda-forge
    pycosat-0.6.3              |py310h5764c6d_1010         108 KB  conda-forge
    python-3.10.6              |h582c2e5_0_cpython        29.0 MB  conda-forge
    python_abi-3.10            |          2_cp310           4 KB  conda-forge
    readline-8.1.2             |       h0f457ee_0         291 KB  conda-forge
    ruamel_yaml-0.15.80        |py310h5764c6d_1007         249 KB  conda-forge
    ------------------------------------------------------------
                                           Total:        35.8 MB

The following NEW packages will be INSTALLED:

  libnsl             conda-forge/linux-64::libnsl-2.0.0-h7f98852_0
  libsqlite          conda-forge/linux-64::libsqlite-3.39.3-h753d276_0
  libuuid            conda-forge/linux-64::libuuid-2.32.1-h7f98852_1000

The following packages will be REMOVED:

  sqlite-3.36.0-h9cd32fc_2

The following packages will be UPDATED:

  ncurses                                    6.2-h58526e2_4 --> 6.3-h27087fc_1
  python                           3.9.7-hb7a2778_3_cpython --> 3.10.6-h582c2e5_0_cpython
  python_abi                                     3.9-2_cp39 --> 3.10-2_cp310
  readline                                   8.1-h46c0cb4_0 --> 8.1.2-h0f457ee_0

The following packages will be DOWNGRADED:

  brotlipy                          0.7.0-py39hb9d737c_1004 --> 0.7.0-py310h5764c6d_1004
  cffi                                1.15.1-py39he91dace_0 --> 1.15.1-py310h255011f_0
  conda                               4.14.0-py39hf3d152e_0 --> 4.14.0-py310hff52083_0
  conda-package-han~                   1.8.1-py39hb9d737c_1 --> 1.8.1-py310h5764c6d_1
  cryptography                        37.0.4-py39hd97740a_0 --> 37.0.4-py310h597c629_0
  libmambapy                          0.25.0-py39hd55135b_2 --> 0.25.0-py310hab0e683_2
  mamba                               0.25.0-py39hfa8f2c8_2 --> 0.25.0-py310hf87f941_2
  pycosat                           0.6.3-py39hb9d737c_1010 --> 0.6.3-py310h5764c6d_1010
  ruamel_yaml                     0.15.80-py39hb9d737c_1007 --> 0.15.80-py310h5764c6d_1007


Proceed ([y]/n)? y


Downloading and Extracting Packages
conda-package-handli | 1015 KB   | ################################################################################################################################################################################################ | 100% 
brotlipy-0.7.0       | 342 KB    | ################################################################################################################################################################################################ | 100% 
cryptography-37.0.4  | 1.6 MB    | ################################################################################################################################################################################################ | 100% 
python-3.10.6        | 29.0 MB   | ################################################################################################################################################################################################ | 100% 
cffi-1.15.1          | 231 KB    | ################################################################################################################################################################################################ | 100% 
pycosat-0.6.3        | 108 KB    | ################################################################################################################################################################################################ | 100% 
libuuid-2.32.1       | 28 KB     | ################################################################################################################################################################################################ | 100% 
python_abi-3.10      | 4 KB      | ################################################################################################################################################################################################ | 100% 
mamba-0.25.0         | 46 KB     | ################################################################################################################################################################################################ | 100% 
libmambapy-0.25.0    | 297 KB    | ################################################################################################################################################################################################ | 100% 
conda-4.14.0         | 1.0 MB    | ################################################################################################################################################################################################ | 100% 
ncurses-6.3          | 1002 KB   | ################################################################################################################################################################################################ | 100% 
readline-8.1.2       | 291 KB    | ################################################################################################################################################################################################ | 100% 
ruamel_yaml-0.15.80  | 249 KB    | ################################################################################################################################################################################################ | 100% 
libnsl-2.0.0         | 31 KB     | ################################################################################################################################################################################################ | 100% 
libsqlite-3.39.3     | 789 KB    | ################################################################################################################################################################################################ | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Retrieving notices: ...working... failed
Traceback (most recent call last):
  File "/opt/tljh/user/lib/python3.9/site-packages/conda/exceptions.py", line 1125, in __call__
  File "/opt/tljh/user/lib/python3.9/site-packages/conda/cli/main.py", line 86, in main_subshell
  File "/opt/tljh/user/lib/python3.9/site-packages/conda/cli/conda_argparse.py", line 93, in do_call
  File "/opt/tljh/user/lib/python3.9/site-packages/conda/notices/core.py", line 75, in wrapper
  File "/opt/tljh/user/lib/python3.9/site-packages/conda/notices/core.py", line 39, in display_notices
  File "/opt/tljh/user/lib/python3.9/site-packages/conda/notices/http.py", line 36, in get_notice_responses
  File "/opt/tljh/user/lib/python3.9/site-packages/conda/notices/http.py", line 39, in <genexpr>
  File "/opt/tljh/user/lib/python3.9/concurrent/futures/_base.py", line 608, in result_iterator
  File "/opt/tljh/user/lib/python3.9/concurrent/futures/_base.py", line 445, in result
  File "/opt/tljh/user/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result
  File "/opt/tljh/user/lib/python3.9/concurrent/futures/thread.py", line 52, in run
  File "/opt/tljh/user/lib/python3.9/site-packages/conda/notices/http.py", line 42, in <lambda>
  File "/opt/tljh/user/lib/python3.9/site-packages/conda/notices/cache.py", line 37, in wrapper
  File "/opt/tljh/user/lib/python3.9/site-packages/conda/notices/http.py", line 58, in get_channel_notice_response
  File "/opt/tljh/user/lib/python3.9/site-packages/requests/sessions.py", line 600, in get
  File "/opt/tljh/user/lib/python3.9/site-packages/requests/sessions.py", line 587, in request
  File "/opt/tljh/user/lib/python3.9/site-packages/requests/sessions.py", line 701, in send
  File "/opt/tljh/user/lib/python3.9/site-packages/requests/adapters.py", line 460, in send
  File "/opt/tljh/user/lib/python3.9/site-packages/requests/adapters.py", line 263, in cert_verify
OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /opt/tljh/user/lib/python3.9/site-packages/certifi/cacert.pem

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/tljh/user/bin/conda", line 13, in <module>
    sys.exit(main())
  File "/opt/tljh/user/lib/python3.9/site-packages/conda/cli/main.py", line 129, in main
  File "/opt/tljh/user/lib/python3.9/site-packages/conda/exceptions.py", line 1413, in conda_exception_handler
  File "/opt/tljh/user/lib/python3.9/site-packages/conda/exceptions.py", line 1128, in __call__
  File "/opt/tljh/user/lib/python3.9/site-packages/conda/exceptions.py", line 1170, in handle_exception
  File "/opt/tljh/user/lib/python3.9/site-packages/conda/exceptions.py", line 1181, in handle_unexpected_exception
  File "/opt/tljh/user/lib/python3.9/site-packages/conda/exceptions.py", line 1251, in print_unexpected_error_report
ModuleNotFoundError: No module named 'conda.cli.main_info'

[mtav]

I just discovered there is a pull request for allowing specifying the python version on install: #784 Not quite a solution to the upgrade problem, but it will certainly help people doing a fresh install, who want a specific python version.