SwarmSpawner, InvalidArgument: Incompatible options have been provided for the bind type mount.

Trying to deploy hub in Docker Swarm on Docker UCP. We are able to to get the main hub up and running, but when we try to spawn the single user notebooks we get the following:

EAE_jupyterhub.1.zluhmt93kq7u@linuxkit-025000000001    |     Traceback (most recent call last):
EAE_jupyterhub.1.zluhmt93kq7u@linuxkit-025000000001    |       File "/opt/conda/lib/python3.6/site-packages/tornado/web.py", line 1592, in _execute
EAE_jupyterhub.1.zluhmt93kq7u@linuxkit-025000000001    |         result = yield result
EAE_jupyterhub.1.zluhmt93kq7u@linuxkit-025000000001    |       File "/opt/conda/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 1052, in get
EAE_jupyterhub.1.zluhmt93kq7u@linuxkit-025000000001    |         await self.spawn_single_user(user)
EAE_jupyterhub.1.zluhmt93kq7u@linuxkit-025000000001    |       File "/opt/conda/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 705, in spawn_single_user
EAE_jupyterhub.1.zluhmt93kq7u@linuxkit-025000000001    |         timedelta(seconds=self.slow_spawn_timeout), finish_spawn_future
EAE_jupyterhub.1.zluhmt93kq7u@linuxkit-025000000001    |       File "/opt/conda/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 626, in finish_user_spawn
EAE_jupyterhub.1.zluhmt93kq7u@linuxkit-025000000001    |         await spawn_future
EAE_jupyterhub.1.zluhmt93kq7u@linuxkit-025000000001    |       File "/opt/conda/lib/python3.6/site-packages/jupyterhub/user.py", line 489, in spawn
EAE_jupyterhub.1.zluhmt93kq7u@linuxkit-025000000001    |         raise e
EAE_jupyterhub.1.zluhmt93kq7u@linuxkit-025000000001    |       File "/opt/conda/lib/python3.6/site-packages/jupyterhub/user.py", line 409, in spawn
EAE_jupyterhub.1.zluhmt93kq7u@linuxkit-025000000001    |         url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
EAE_jupyterhub.1.zluhmt93kq7u@linuxkit-025000000001    |       File "/opt/conda/lib/python3.6/site-packages/dockerspawner/dockerspawner.py", line 691, in start
EAE_jupyterhub.1.zluhmt93kq7u@linuxkit-025000000001    |         obj = yield self.create_object()
EAE_jupyterhub.1.zluhmt93kq7u@linuxkit-025000000001    |       File "/opt/conda/lib/python3.6/site-packages/dockerspawner/swarmspawner.py", line 173, in create_object
EAE_jupyterhub.1.zluhmt93kq7u@linuxkit-025000000001    |         mounts=self.mounts,
EAE_jupyterhub.1.zluhmt93kq7u@linuxkit-025000000001    |       File "/opt/conda/lib/python3.6/site-packages/dockerspawner/swarmspawner.py", line 110, in mounts
EAE_jupyterhub.1.zluhmt93kq7u@linuxkit-025000000001    |         for host_loc, vol in self.volume_binds.items()
EAE_jupyterhub.1.zluhmt93kq7u@linuxkit-025000000001    |       File "/opt/conda/lib/python3.6/site-packages/dockerspawner/swarmspawner.py", line 110, in <listcomp>
EAE_jupyterhub.1.zluhmt93kq7u@linuxkit-025000000001    |         for host_loc, vol in self.volume_binds.items()
EAE_jupyterhub.1.zluhmt93kq7u@linuxkit-025000000001    |       File "/opt/conda/lib/python3.6/site-packages/docker/types/services.py", line 237, in __init__
EAE_jupyterhub.1.zluhmt93kq7u@linuxkit-025000000001    |         'Incompatible options have been provided for the bind '
EAE_jupyterhub.1.zluhmt93kq7u@linuxkit-025000000001    |     docker.errors.InvalidArgument: Incompatible options have been provided for the bind type mount.

jupyterhub-config had the following lines to handle volumes

notebook_dir = '/home/jovyan'
c.SwarmSpawner.notebook_dir = notebook_dir
c.SwarmSpawner.volumes = {'/mnt/our_shared_path': notebook_dir, 'mode': 'rw'}
c.SwarmSpawner.volume_driver = None

Tried commenting out the ‘SwarmSpawner.volumes’ line but we still see the same error as above. According to binds in swarmspawner.py (https://github.com/jupyterhub/dockerspawner/blob/master/dockerspawner/swarmspawner.py#L99), if len(self.volume_binds) == 0 it should just return [] and there should be no error. We do not create a mount in docker-compose or manually on the host. Using Jupyterhub version 0.9.4. Haven’t been able to find a way around this issue.

[vlizanae]

@wlohman-surfsara I tried this by overwriting the mounts in the config file:

from dockerspawner import SwarmSpawner
from docker.types import Mount

class SwarmSpawner2(SwarmSpawner):
    @property
    def mounts(self):
        if len(self.volume_binds):
            driver = self.mount_driver_config
            return [
                Mount(
                    target=vol["bind"],
                    source=host_loc,
                    type="bind",
                    read_only=vol["mode"] == "ro",
                    driver_config=None,
                )
                for host_loc, vol in self.volume_binds.items()
            ]

        else:
            return []

c.JupyterHub.spawner_class = SwarmSpawner2

c.SwarmSpawner2.network_name = os.environ['DOCKER_NETWORK_NAME']
c.SwarmSpawner2.extra_host_config = {'network_mode': os.environ['DOCKER_NETWORK_NAME']}

c.SwarmSpawner2.image = 'registry:5000/jupyterhub_singleuser'

c.SwarmSpawner2.notebook_dir = '/home/jovyan'
c.SwarmSpawner2.volumes = {'/data/home/{username}': '/home/jovyan'}
c.SwarmSpawner2.remove_containers = True

When I comment the line with c.SwarmSpawner2.volumes works (without persistency obviously) but when it is on all I get is

Spawner failed to start [status=0]

No logs in the spawned service also, which is there but with 0 replicas, I tried to escalate it but it fails.

[vlizanae]

It works now, it was a permissions issue, but this is definitely a bug, should be labeled as such.