Error mounting /tmp/airflowtmp... with remote docker

Apache Airflow version: v2.1.0

Environment:

• Cloud provider or hardware configuration: ec2 t3a.medium
• OS (e.g. from /etc/os-release): Ubuntu 18.04.5 LTS
• Kernel (e.g. uname -a): 5.4.0-1051-aws
• Install tools: sudo pip3 install apache-airflow[mysql,ssh,docker,amazon]
• Others: python 3.6.9

What happened:

Task fails with error:

docker.errors.APIError: 400 Client Error for http://192.168.1.50:2375/v1.41/containers/create: 
Bad Request ("invalid mount config for type "bind": bind source path does not exist: /tmp/airflowtmp7naq_r53")

How to reproduce it:

Create an separate EC2 instance and forward the docker daemon:

sudo mkdir -p /etc/systemd/system/docker.service.d
sudo touch /etc/systemd/system/docker.service.d/options.conf
echo -e """
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H unix:// -H tcp://0.0.0.0:2375
""" >> /etc/systemd/system/docker.service.d/options.conf
sudo systemctl daemon-reload
sudo systemctl restart docker

Create dag with DockerOperator

DockerOperator(
        task_id="run_image",
        docker_url="tcp://192.168.1.50:2375",
        image="ubuntu:latest",
        dag=dag,
    )

Run the DAG.

Anything else we need to know:

To me it looks like the DockerOperator is creating a temporary directory locally and tries to bind it to the container. However as this is a remote container the directory doesn’t exist. here is the code part:

class DockerOperator(BaseOperator):
    ...

    def _run_image(self) -> Optional[str]:
        """Run a Docker container with the provided image"""
        self.log.info('Starting docker container from image %s', self.image)

        with TemporaryDirectory(prefix='airflowtmp', dir=self.host_tmp_dir) as host_tmp_dir:
            if not self.cli:
                raise Exception("The 'cli' should be initialized before!")
            tmp_mount = Mount(self.tmp_dir, host_tmp_dir, "bind")
            self.container = self.cli.create_container(
                command=self.format_command(self.command),
                name=self.container_name,
                environment={**self.environment, **self._private_environment},
                host_config=self.cli.create_host_config(
                    auto_remove=False,
                    mounts=self.mounts + [tmp_mount],
                    network_mode=self.network_mode,
                    shm_size=self.shm_size,
                    dns=self.dns,
                    dns_search=self.dns_search,
                    cpu_shares=int(round(self.cpus * 1024)),
                    mem_limit=self.mem_limit,
                    cap_add=self.cap_add,
                    extra_hosts=self.extra_hosts,
                    privileged=self.privileged,
                ),
                image=self.image,
                user=self.user,
                entrypoint=self.format_command(self.entrypoint),
                working_dir=self.working_dir,
                tty=self.tty,
            )

I see no way of disabling this behavior without some major patching.

How are you guys using remote docker daemons? Is this a use case? Would it be possible to implement something to allow that?

[potiuk]

Ah. OK. I see Indeed. I thin kit was because we have changed how “Mount” works and while it was not failing before, it did not work properly either (the tmp volume was not mounted). I see then how it was not detected before. Good call @sudohainguyen !

[akki]

I don’t have a complete proof for this yet (I might have to look into Docker’s code for that), but from the docs (and the error message), it seems that we might be using the wrong type.

Is it possible for anyone here to change this line to tmp_mount = Mount(self.tmp_dir, host_tmp_dir, "volume") and see if that fixes the issue?

More details in this comment - https://github.com/apache/airflow/pull/16932#discussion_r668269023