Flow Deployment Failure - Prefect Docker Agent Running in Another Docker Container

Logging this issue per request in Prefect slack channel

Description

A clear description of the bug

When running a prefect docker agent within another docker container (docker within docker) - I get the following error during flow deployment.

I didn’t have this issue on an older version of the docker daemon - 1.13.

[2020-06-10 23:02:59,073] INFO - agent | Successfully pulled image <ecr_image_name>… [2020-06-10 23:02:59,180] ERROR - agent | Logging platform error for flow run ee3bb337-ad78-466a-a774-6e67f467aedc [2020-06-10 23:02:59,500] ERROR - agent | Error while deploying flow: APIError(HTTPError(’400 Client Error: Bad Request for url: http+docker://localhost/v1.40/containers/create’))

Expected Behavior

What did you expect to happen instead?

The flow deploys successfully.

Reproduction

A minimal example that exhibits the behavior.

Base Dockerfile (will launch prefect docker agent on start)

FROM python:3.7
ENV TZ=America/Denver

RUN pip install prefect awscli
RUN apt-get update
RUN apt-get install docker.io -y
COPY entrypoint.sh /opt/
COPY local.entrypoint.sh /opt/

RUN chmod +x /opt/entrypoint.sh

ENTRYPOINT ["/opt/entrypoint.sh"]

entrypoint.sh script contents:

service docker start prefect agent start docker -l affinity_label -t <token>

Launch the base docker container with this: docker run --name $container $image -v /var/run/docker.sock:/var/run/docker.sock --privileged

Then, register and deploy any basic flow to this agent.

Additional Info: I was able to successfully deploy a new docker container within the base docker container via:

import docker
c = docker.APIClient(version="auto")
container = c.create_container("my_image", command="echo 1")
c.start(container=container.get("Id"))

Environment

Any additional information about your environment

Optionally run prefect diagnostics from the command line and paste the information here

[pedromachados]

I am seeing a similar issue running Docker version 19.03.13, build 4484 on Ubuntu (WSL 2). Running the agent with --no-docker-interface solved it for me as well.

[lauralorenz]

Hi @cvrooman, 100 years later and a couple flips between linux and macos I was able to reproduce your issue, and also get an agent running in editable mode so I could drop into pdb easy. It looks like the underlying docker client error which is getting swallowed somewhere is actually:

-> return self.create_container_from_config(config, name)
(Pdb) n
docker.errors.APIError: 400 Client Error: Bad Request ("invalid IP address in add-host: """)

which excitingly (in a morbid way) I think is also related to another active bug, https://github.com/PrefectHQ/prefect/issues/2960.

A few more pdb logs that are helpful:

# stepping through and seeing that the extra hosts didn't get set the way we expect
> /home/prefect/src/prefect/agent/docker/agent.py(359)deploy_flow()
-> if sys.platform.startswith("linux") and self.docker_interface:
(Pdb) n
> /home/prefect/src/prefect/agent/docker/agent.py(360)deploy_flow()
-> docker_internal_ip = get_docker_ip()
(Pdb) n
> /home/prefect/src/prefect/agent/docker/agent.py(361)deploy_flow()
-> host_config.update(extra_hosts={"host.docker.internal": docker_internal_ip})
(Pdb) n
> /home/prefect/src/prefect/agent/docker/agent.py(363)deploy_flow()
-> networking_config = None
(Pdb) host_config
{'auto_remove': True, 'extra_hosts': {'host.docker.internal': ''}}

# deeper in the docker client, seeing that this host config is still weird
-> def create_container_config(self, *args, **kwargs):
(Pdb) n
> /usr/local/lib/python3.7/site-packages/docker/api/container.py(433)create_container_config()
-> return ContainerConfig(self._version, *args, **kwargs)
(Pdb) args
self = <docker.api.client.APIClient object at 0x7f4091491110>
args = ('my-first-flow:2020-07-23t16-39-59-453323-00-00', 'prefect execute cloud-flow', None, None, False, False, False, None, ['PREFECT__CLOUD__API=https://api.prefect.io', "PREFECT__CLOUD__AGENT__LABELS=['affinity_label']", 'PREFECT__CONTEXT__FLOW_RUN_ID=b96d48ba-36b4-45c7-89b2-94c680d1bdaf', 'PREFECT__CONTEXT__FLOW_ID=5a3e36aa-715a-4fa8-bda3-9b32e22818b0', 'PREFECT__CLOUD__USE_LOCAL_SECRETS=false', 'PREFECT__LOGGING__LOG_TO_CLOUD=true', 'PREFECT__LOGGING__LEVEL=DEBUG', 'PREFECT__ENGINE__FLOW_RUNNER__DEFAULT_CLASS=prefect.engine.cloud.CloudFlowRunner', 'PREFECT__ENGINE__TASK_RUNNER__DEFAULT_CLASS=prefect.engine.cloud.CloudTaskRunner'], [], False, None, None, None, {'NetworkMode': 'default', 'ExtraHosts': ['host.docker.internal:'], 'AutoRemove': True}, None, None, None, None, None, None, None)
kwargs = {}
(Pdb) 

Continuing to investigate, now in concert with https://github.com/PrefectHQ/prefect/issues/2960!