Spawning error on Docker Desktop Edge

Bug description

When testing dockerspawner 0.11.1 with JupyterHub (on Docker Desktop for Mac Edge 2.3.7.0, spawning failed with some errors. The same setup worked fine with Docker Desktop for Mac Stable 2.3.0.5. I understand Edge is supposed to be unstable, but I’m reporting just in case developers might get some insights (i.e. API change?) from this.

Expected behaviour

Spawning a new image without error.

Actual behaviour

Spawning failed with error.

[I 2020-09-19 18:49:09.020 JupyterHub app:2721] JupyterHub is now running at http://:8000
[I 2020-09-19 18:49:17.682 JupyterHub log:181] 302 GET / -> /hub/ (@::ffff:172.18.0.1) 0.99ms
[I 2020-09-19 18:49:17.716 JupyterHub log:181] 302 GET /hub/ -> /hub/spawn (tomyun@::ffff:172.18.0.1) 21.32ms
[E 2020-09-19 18:49:17.878 JupyterHub user:690] Unhandled error starting tomyun's server: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
[E 2020-09-19 18:49:17.925 JupyterHub user:700] Failed to cleanup tomyun's server that failed to start
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 698, in spawn
        await self.stop(spawner.name)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 795, in stop
        status = await spawner.poll()
      File "/usr/local/lib/python3.8/dist-packages/dockerspawner/dockerspawner.py", line 755, in poll
        container = yield self.get_object()
      File "/usr/local/lib/python3.8/dist-packages/dockerspawner/dockerspawner.py", line 779, in get_object
        obj = yield self.docker("inspect_%s" % self.object_type, self.object_name)
      File "/usr/lib/python3.8/concurrent/futures/_base.py", line 432, in result
        return self.__get_result()
      File "/usr/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
        raise self._exception
      File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
        result = self.fn(*self.args, **self.kwargs)
      File "/usr/local/lib/python3.8/dist-packages/dockerspawner/dockerspawner.py", line 743, in _docker
        return m(*args, **kwargs)
      File "/usr/local/lib/python3.8/dist-packages/docker/utils/decorators.py", line 19, in wrapped
        return f(self, resource_id, *args, **kwargs)
      File "/usr/local/lib/python3.8/dist-packages/docker/api/container.py", line 771, in inspect_container
        self._get(self._url("/containers/{0}/json", container)), True
      File "/usr/local/lib/python3.8/dist-packages/docker/utils/decorators.py", line 46, in inner
        return f(self, *args, **kwargs)
      File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 228, in _get
        return self.get(url, **self._set_request_timeout(kwargs))
      File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 543, in get
        return self.request('GET', url, **kwargs)
      File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 530, in request
        resp = self.send(prep, **send_kwargs)
      File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 685, in send
        r.content
      File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 829, in content
        self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
      File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 754, in generate
        raise ChunkedEncodingError(e)
    requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

[W 2020-09-19 18:49:17.929 JupyterHub web:1786] 500 GET /hub/spawn (::ffff:172.18.0.1): Error in Authenticator.pre_spawn_start: ChunkedEncodingError ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
[E 2020-09-19 18:49:17.963 JupyterHub log:173] {
      "X-Forwarded-Host": "localhost:8000",
      "X-Forwarded-Proto": "http",
      "X-Forwarded-Port": "8000",
      "X-Forwarded-For": "::ffff:172.18.0.1",
      "Cookie": "jupyterhub-hub-login=[secret]; jupyterhub-session-id=[secret]",
      "Accept-Language": "en-US,en;q=0.9",
      "Accept-Encoding": "gzip, deflate, br",
      "Sec-Fetch-Dest": "document",
      "Sec-Fetch-User": "?1",
      "Sec-Fetch-Mode": "navigate",
      "Sec-Fetch-Site": "none",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36",
      "Upgrade-Insecure-Requests": "1",
      "Connection": "close",
      "Host": "localhost:8000"
    }
[E 2020-09-19 18:49:17.963 JupyterHub log:181] 500 GET /hub/spawn (tomyun@::ffff:172.18.0.1) 236.28ms

How to reproduce

1. Set up JupyterHub with dockerspawner. The base image I used for JupyterHub is 1.2. I followed instructions mostly from this.
2. Run JupyterHub
3. Log in with any username (dummy auth)
4. See error

Your personal set up

• OS: macOS 10.15
• Version: JupyterHub 1.2 (docker), dockerspawner 0.11.1, Docker Desktop for Mac Edge 2.3.7.0
• Configuration: described above

[bjornandre]

Another workaround is to use the non-proxied socket at /var/run/docker.sock.raw. That seems to do the trick for me. So in @tomyun’s example:

docker run --rm -it -v /var/run/docker.sock.raw:/var/run/docker.sock --net jupyterhub --name jupyterhub -p8000:8000 hub

[bjornandre]

Possibly related to this: https://github.com/docker/for-mac/issues/5072