docker from_env and pull is broken on mac

Unusual issue here - code that was working on Thursday is no longer working today, so this might be tied into some behind-the-scenes update to Docker Desktop. On that note, I’ve also tried with explicit Docker Desktop versions 4.13.0, 4.12.0, 4.11.1 (and used the clean uninstaller to remove the VMs and temp folders left over) and all report the same issue.

Docker-desktop 4.10.1 does not produce an error, and it is the version I am now migrating to.

Reproduction:

1. conda create -n docker python=3.10 && conda activate docker
2. pip install docker
3. python -c "import docker; client = docker.from_env()"

The final command fails.

Traceback (most recent call last):
  File "/Users/sh/miniconda3/envs/docker/lib/python3.10/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/Users/sh/miniconda3/envs/docker/lib/python3.10/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/Users/sh/miniconda3/envs/docker/lib/python3.10/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/Users/sh/miniconda3/envs/docker/lib/python3.10/site-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/Users/sh/miniconda3/envs/docker/lib/python3.10/site-packages/requests/sessions.py", line 600, in get
    return self.request("GET", url, **kwargs)
  File "/Users/sh/miniconda3/envs/docker/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/sh/miniconda3/envs/docker/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/Users/sh/miniconda3/envs/docker/lib/python3.10/site-packages/requests/adapters.py", line 547, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/sh/miniconda3/envs/docker/lib/python3.10/site-packages/docker/client.py", line 96, in from_env
    return cls(
  File "/Users/sh/miniconda3/envs/docker/lib/python3.10/site-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
  File "/Users/sh/miniconda3/envs/docker/lib/python3.10/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/Users/sh/miniconda3/envs/docker/lib/python3.10/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

By specifying version="1.35" when making a client, it will not fail, but then any subsequent client.images.pull will fail regardless:

python -c "import docker; client = docker.from_env(version='1.35'); client.images.pull('docker:latest');"

  File "/Users/sh/miniconda3/envs/py3106/lib/python3.10/site-packages/docker/models/images.py", line 465, in pull
    pull_log = self.client.api.pull(
  File "/Users/sh/miniconda3/envs/py3106/lib/python3.10/site-packages/docker/api/image.py", line 424, in pull
    response = self._post(
  File "/Users/sh/miniconda3/envs/py3106/lib/python3.10/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/Users/sh/miniconda3/envs/py3106/lib/python3.10/site-packages/docker/api/client.py", line 233, in _post
    return self.post(url, **self._set_request_timeout(kwargs))
  File "/Users/sh/miniconda3/envs/py3106/lib/python3.10/site-packages/requests/sessions.py", line 577, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/Users/sh/miniconda3/envs/py3106/lib/python3.10/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/sh/miniconda3/envs/py3106/lib/python3.10/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/Users/sh/miniconda3/envs/py3106/lib/python3.10/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

I can confirm docker is running during these as well. Screenshot attached.

I separate team member has been able to reproduce this, but on their m1 hardware (I ran it on my x86 based mac).

[mndeveci]

I was able to track down this issue. With latest release of docker, the context of the client is changed from default to desktop-linux which uses different endpoint and therefore breaks the docker client.

Run docker info to see which context client uses;

$ docker info
Client:
 Context:    desktop-linux
...

And the run docker context ls to find out its endpoint configuration;

$ docker context ls
NAME                TYPE                DESCRIPTION                               DOCKER ENDPOINT                                  KUBERNETES ENDPOINT   ORCHESTRATOR
default             moby                Current DOCKER_HOST based configuration   unix:///var/run/docker.sock                                            swarm
desktop-linux *     moby                                                          unix:///Users/ec2-user/.docker/run/docker.sock

Then setting DOCKER_HOST to the endpoint that you see for the default context should resolve the issue.

But the problem is, how can we know this endpoint configuration, if we can’t connect to docker daemon?

Maintainers, we need your feedback to resolve this issue.

[fabiosanger]

Docker desktop version is 4.13.1 but i still get the same error