question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

key_error runtime_dir

See original GitHub issue

Hi,

I’m running jupyterhub on a cluster submit node dedicated for it and use the batchspawner plus profilespawner spinning up notebooks for students. I am running into some issues with jupyterhub 0.7.2 and batchspawner 02c0b40d614cea78d08882ade18b227637e52768. We use local accounts. Here is my jupyterhub_config.py on the head node:

c.JupyterHub.hub_ip = '0.0.0.0'
c.JupyterHub.cookie_max_age_days = 8
c = get_config()
c.JupyterHub.spawner_class = 'batchspawner.SlurmSpawner'
c.Spawner.http_timeout = 120
c.SlurmSpawner.req_nprocs = '2'
c.SlurmSpawner.req_runtime = '12:00:00'
c.SlurmSpawner.batch_script = '''#!/bin/bash
#SBATCH --partition={partition}
#SBATCH --time={runtime}
#SBATCH --output={homedir}/jupyterhub_slurmspawner_%j.log
#SBATCH --job-name=spawner-jupyterhub
#SBATCH --workdir={homedir}
#SBATCH --mem={memory}
#SBATCH --export={keepvars}
#SBATCH --uid={username}
#SBATCH --get-user-env=L
#SBATCH {options}


module load courses/env

which jupyterhub-singleuser
{cmd}
'''
c.JupyterHub.spawner_class = 'wrapspawner.ProfilesSpawner'
c.Spawner.http_timeout = 120
c.ProfilesSpawner.profiles = [
   ( "Local server", 'local', 'jupyterhub.spawner.LocalProcessSpawner', {'ip':'0.0.0.0'} ),
   ('cluster Interactive GPU - 6 cores + Nvidia GTX 1080,  16 GB, 8 hours',
    'interactive-gpu',
    'batchspawner.SlurmSpawner',
      dict(req_nprocs='2', req_partition='gpu', req_runtime='8:00:00')
   )
   ]

c.Spawner.env_keep = ['PATH', 'LD_LIBRARY_PATH', 'PYTHONPATH', 'CONDA_ROOT', 'CONDA_DEFAULT_ENV', 'VIRTUAL_ENV', 'LANG', 'LC_ALL']

The problem is, that I’m getting an error message in the slurm job’s log that I don’t understand:

/home/courses/sw/apps/jupyterhub/0.7.2/bin/jupyterhub-singleuser
Traceback (most recent call last):
  File "/sw/apps/python/3.5.1/lib/python3.5/site-packages/traitlets/traitlets.py", line 528, in get
    value = obj._trait_values[self.name]
KeyError: 'runtime_dir'

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/courses/sw/apps/jupyterhub/0.7.2/bin/jupyterhub-singleuser", line 6, in <module>
    main()
  File "/home/courses/sw/apps/jupyterhub/0.7.2/lib/python3.5/site-packages/jupyterhub/singleuser.py", line 322, in main
    return SingleUserNotebookApp.launch_instance(argv)
  File "/home/courses/sw/lib/python3.5/site-packages/jupyter_core/application.py", line 267, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/sw/apps/python/3.5.1/lib/python3.5/site-packages/traitlets/config/application.py", line 657, in launch_instance
    app.initialize(argv)
  File "<decorator-gen-7>", line 2, in initialize
  File "/sw/apps/python/3.5.1/lib/python3.5/site-packages/traitlets/config/application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "/home/courses/sw/lib/python3.5/site-packages/notebook/notebookapp.py", line 1294, in initialize
    self.init_configurables()
  File "/home/courses/sw/lib/python3.5/site-packages/notebook/notebookapp.py", line 1033, in init_configurables
    connection_dir=self.runtime_dir,
  File "/sw/apps/python/3.5.1/lib/python3.5/site-packages/traitlets/traitlets.py", line 556, in __get__
    return self.get(obj, cls)
  File "/sw/apps/python/3.5.1/lib/python3.5/site-packages/traitlets/traitlets.py", line 535, in get
    value = self._validate(obj, dynamic_default())
  File "/home/courses/sw/lib/python3.5/site-packages/jupyter_core/application.py", line 99, in _runtime_dir_default
    ensure_dir_exists(rd, mode=0o700)
  File "/sw/apps/python/3.5.1/lib/python3.5/site-packages/ipython_genutils/path.py", line 167, in ensure_dir_exists
    os.makedirs(path, mode=mode)
  File "/sw/apps/python/3.5.1/lib/python3.5/os.py", line 231, in makedirs
    makedirs(head, mode, exist_ok)
  File "/sw/apps/python/3.5.1/lib/python3.5/os.py", line 241, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/run/user/8900'

I am not sure where runtime_dir is coming from. I checked in the code bases of jupyterhub, batchspawner and wrapspawner.

Any ideas would be appreciated.

PS. It would be nice if there could be a tested jupyterhub_config.py as part of this repo.

Issue Analytics

  • State:closed
  • Created 6 years ago
  • Comments:14 (8 by maintainers)

github_iconTop GitHub Comments

3reactions
rkdarstcommented, Apr 21, 2018

I’d say, even in an HPC environment, we should take security seriously.

What about adding unset XDG_RUNTIME_DIR in the batch script right before the jupyterhub-singleuser command?

0reactions
psteinbcommented, Aug 17, 2017

solved it … apparently the following 2 lines contained in the slurm batch script cause the system quite some headaches:

#SBATCH --export={keepvars}
#SBATCH --get-user-env=L

jupyterhub-singleuser does report now that

[W 2017-08-17 09:43:27.923 steinbac login:225] All authentication is disabled.  Anyone who can connect to this server will be able to run code.

But I feel that this is justifyable in an HPC environment. Feel free to disagree or chime in to add more details what removing the 2 lines above results in.

Read more comments on GitHub >

github_iconTop Results From Across the Web

python - Jupyter notebook permission error - Stack Overflow
the solution is run the following command on the terminal as a normal user. $ export XDG_RUNTIME_DIR="" then start the jupyter $jupyter notebook....
Read more >
jupyterhub/jupyterhub - Gitter
KeyError : 'runtime_dir'. During handling of the above exception, another exception occurred: ... connection_dir=self.runtime_dir,
Read more >
Permission issue for /run/user/0/jupyter when ... - Google Groups
KeyError : 'runtime_dir' During handling of the above exception, another exception occurred: Traceback (most recent call last):
Read more >
Jupyter notebook is not running [PermissionError: [Errno 13 ...
I have found the answer 1. first open your terminal and uninstall the jupyter notebook pip uninstall jupyter_core
Read more >
jupyter notebook, KeyError: 'runtime_dir',[Errno 13] Permission ...
KeyError : 'runtime_dir'. During handling of the above exception, another exception occurred: Traceback (most recent call last):.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found