404 on containers.create for valid images

tested under:

Python 3.6.5 Docker version 17.12.1-ce, build 7390fc6 docker 3.4.1

and

Python 3.7.0 Docker version 18.06.0-ce, build 0ffa825 docker 3.4.1

When running

>>> import docker as dockerpy
>>> _client = dockerpy.from_env()
>>> container = _client.containers.create(image='tomochain/infra-telegraf:devnet', name='telegraf')
Traceback (most recent call last):
  File "/Users/etienne/Library/Caches/pypoetry/virtualenvs/tmn-py3.7/lib/python3.7/site-packages/docker/api/client.py", line 229, in _raise_for_status
    response.raise_for_status()
  File "/Users/etienne/Library/Caches/pypoetry/virtualenvs/tmn-py3.7/lib/python3.7/site-packages/requests/models.py", line 939, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.35/containers/create?name=telegraf

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/etienne/Library/Caches/pypoetry/virtualenvs/tmn-py3.7/lib/python3.7/site-packages/docker/models/containers.py", line 824, in create
    resp = self.client.api.create_container(**create_kwargs)
  File "/Users/etienne/Library/Caches/pypoetry/virtualenvs/tmn-py3.7/lib/python3.7/site-packages/docker/api/container.py", line 411, in create_container
    return self.create_container_from_config(config, name)
  File "/Users/etienne/Library/Caches/pypoetry/virtualenvs/tmn-py3.7/lib/python3.7/site-packages/docker/api/container.py", line 422, in create_container_from_config
    return self._result(res, True)
  File "/Users/etienne/Library/Caches/pypoetry/virtualenvs/tmn-py3.7/lib/python3.7/site-packages/docker/api/client.py", line 235, in _result
    self._raise_for_status(response)
  File "/Users/etienne/Library/Caches/pypoetry/virtualenvs/tmn-py3.7/lib/python3.7/site-packages/docker/api/client.py", line 231, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/Users/etienne/Library/Caches/pypoetry/virtualenvs/tmn-py3.7/lib/python3.7/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error: Not Found ("No such image: tomochain/infra-telegraf:devnet")

If I use the docker client:

➜ docker pull tomochain/infra-telegraf:devnet
devnet: Pulling from tomochain/infra-telegraf
55cbf04beb70: Downloading [===============>                                   ]  14.27MB/45.31MB
1607093a898c: Downloading [=====================>                             ]  4.586MB/10.74MB
9a8ea045c926: Download complete
9dbf507f8efa: Downloading [==>                                                ]  817.9kB/15.96MB
b7bbc4db97cd: Waiting
b8898b2dd4b3: Waiting
595512eec936: Waiting
d4bd43885499: Waiting

Works fine. And after that there will be no exceptions as the images are already present locally It also works with the images.pull function:

>>> client.images.pull('tomochain/infra-telegraf:devnet')
<Image: 'tomochain/infra-telegraf:devnet'>

[mitar]

So we have noticed that this happens when running on disk-constrained environment where the Docker image pulled is larger than available space. Then the image pulling in fact fails so when it attempts to do the next step (creating container) that fails with 404. Why exactly pulling does not fail with some exception first is strange. Calling docker pull using a subprocess first shows the error fine.

This is example CI job which fails this way because on GitLab CI workers have limited space and we are trying to run a 11 GB large image: https://gitlab.com/bhshah/primitives/-/jobs/1695097123

[juledwar]

I’m using 5.0.0 and it still happens randomly.