key_error runtime_dir

Hi,

I’m running jupyterhub on a cluster submit node dedicated for it and use the batchspawner plus profilespawner spinning up notebooks for students. I am running into some issues with jupyterhub 0.7.2 and batchspawner 02c0b40d614cea78d08882ade18b227637e52768. We use local accounts. Here is my jupyterhub_config.py on the head node:

c.JupyterHub.hub_ip = '0.0.0.0'
c.JupyterHub.cookie_max_age_days = 8
c = get_config()
c.JupyterHub.spawner_class = 'batchspawner.SlurmSpawner'
c.Spawner.http_timeout = 120
c.SlurmSpawner.req_nprocs = '2'
c.SlurmSpawner.req_runtime = '12:00:00'
c.SlurmSpawner.batch_script = '''#!/bin/bash
#SBATCH --partition={partition}
#SBATCH --time={runtime}
#SBATCH --output={homedir}/jupyterhub_slurmspawner_%j.log
#SBATCH --job-name=spawner-jupyterhub
#SBATCH --workdir={homedir}
#SBATCH --mem={memory}
#SBATCH --export={keepvars}
#SBATCH --uid={username}
#SBATCH --get-user-env=L
#SBATCH {options}


module load courses/env

which jupyterhub-singleuser
{cmd}
'''
c.JupyterHub.spawner_class = 'wrapspawner.ProfilesSpawner'
c.Spawner.http_timeout = 120
c.ProfilesSpawner.profiles = [
   ( "Local server", 'local', 'jupyterhub.spawner.LocalProcessSpawner', {'ip':'0.0.0.0'} ),
   ('cluster Interactive GPU - 6 cores + Nvidia GTX 1080,  16 GB, 8 hours',
    'interactive-gpu',
    'batchspawner.SlurmSpawner',
      dict(req_nprocs='2', req_partition='gpu', req_runtime='8:00:00')
   )
   ]

c.Spawner.env_keep = ['PATH', 'LD_LIBRARY_PATH', 'PYTHONPATH', 'CONDA_ROOT', 'CONDA_DEFAULT_ENV', 'VIRTUAL_ENV', 'LANG', 'LC_ALL']

The problem is, that I’m getting an error message in the slurm job’s log that I don’t understand:

/home/courses/sw/apps/jupyterhub/0.7.2/bin/jupyterhub-singleuser
Traceback (most recent call last):
  File "/sw/apps/python/3.5.1/lib/python3.5/site-packages/traitlets/traitlets.py", line 528, in get
    value = obj._trait_values[self.name]
KeyError: 'runtime_dir'

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/courses/sw/apps/jupyterhub/0.7.2/bin/jupyterhub-singleuser", line 6, in <module>
    main()
  File "/home/courses/sw/apps/jupyterhub/0.7.2/lib/python3.5/site-packages/jupyterhub/singleuser.py", line 322, in main
    return SingleUserNotebookApp.launch_instance(argv)
  File "/home/courses/sw/lib/python3.5/site-packages/jupyter_core/application.py", line 267, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/sw/apps/python/3.5.1/lib/python3.5/site-packages/traitlets/config/application.py", line 657, in launch_instance
    app.initialize(argv)
  File "<decorator-gen-7>", line 2, in initialize
  File "/sw/apps/python/3.5.1/lib/python3.5/site-packages/traitlets/config/application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "/home/courses/sw/lib/python3.5/site-packages/notebook/notebookapp.py", line 1294, in initialize
    self.init_configurables()
  File "/home/courses/sw/lib/python3.5/site-packages/notebook/notebookapp.py", line 1033, in init_configurables
    connection_dir=self.runtime_dir,
  File "/sw/apps/python/3.5.1/lib/python3.5/site-packages/traitlets/traitlets.py", line 556, in __get__
    return self.get(obj, cls)
  File "/sw/apps/python/3.5.1/lib/python3.5/site-packages/traitlets/traitlets.py", line 535, in get
    value = self._validate(obj, dynamic_default())
  File "/home/courses/sw/lib/python3.5/site-packages/jupyter_core/application.py", line 99, in _runtime_dir_default
    ensure_dir_exists(rd, mode=0o700)
  File "/sw/apps/python/3.5.1/lib/python3.5/site-packages/ipython_genutils/path.py", line 167, in ensure_dir_exists
    os.makedirs(path, mode=mode)
  File "/sw/apps/python/3.5.1/lib/python3.5/os.py", line 231, in makedirs
    makedirs(head, mode, exist_ok)
  File "/sw/apps/python/3.5.1/lib/python3.5/os.py", line 241, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/run/user/8900'

I am not sure where runtime_dir is coming from. I checked in the code bases of jupyterhub, batchspawner and wrapspawner.

Any ideas would be appreciated.

PS. It would be nice if there could be a tested jupyterhub_config.py as part of this repo.

[rkdarst]

I’d say, even in an HPC environment, we should take security seriously.

What about adding unset XDG_RUNTIME_DIR in the batch script right before the jupyterhub-singleuser command?

[psteinb]

solved it … apparently the following 2 lines contained in the slurm batch script cause the system quite some headaches:

#SBATCH --export={keepvars}
#SBATCH --get-user-env=L

jupyterhub-singleuser does report now that

[W 2017-08-17 09:43:27.923 steinbac login:225] All authentication is disabled.  Anyone who can connect to this server will be able to run code.

But I feel that this is justifyable in an HPC environment. Feel free to disagree or chime in to add more details what removing the 2 lines above results in.