Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

DockerOperator fails with HTTPError: 400 Client Error

See original GitHub issue

Apache Airflow version: 2.1.0

Kubernetes version (if you are using kubernetes) (use kubectl version): -

Environment:

  • Cloud provider or hardware configuration: OpenStack Victoria
  • OS (e.g. from /etc/os-release): Ubuntu 20.04.2 LTS
  • Kernel (e.g. uname -a): Linux 5.4.0-1041-kvm
  • Install tools: pip
  • Others:
    • Docker version 20.10.7, build f0df350
    • docker-py 5.0.0

What happened:

Created a DAG with basic tasks in it as defined hereafter. DummyOperator --> DockerOperator --> DummyOperator

DockerOperator Instance:

DockerOperator(
        task_id="test",
        image="debian:sid-slim",
        command="hostname",
        mounts=[Mount("/eodc", "/eodc", type="bind")]
    )

What you expected to happen:

Pull image, run container and execute hostname in it.

How to reproduce it:

Setup Airflow 2.1.0 and latest Docker provider

Anything else we need to know:

Task Log Output:

[2021-07-01 10:36:39,537] {base_task_runner.py:115} INFO - Job 11: Subtask test /opt/airflow/lib/python3.8/site-packages/airflow/configuration.py:346 DeprecationWarning: The hide_sensitive_variable_fields option in [admin] has been moved to the hide_sensitive_var_conn_fields option in [core] - the old setting has been used, but please update your config.
[2021-07-01 10:36:39,856] {base_task_runner.py:115} INFO - Job 11: Subtask test /opt/airflow/lib/python3.8/site-packages/airflow/configuration.py:346 DeprecationWarning: The default_queue option in [celery] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config.
[2021-07-01 10:36:40,032] {base_task_runner.py:115} INFO - Job 11: Subtask test /opt/airflow/lib/python3.8/site-packages/airflow/configuration.py:346 DeprecationWarning: The default_queue option in [celery] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config.
[2021-07-01 10:36:40,243] {base_task_runner.py:115} INFO - Job 11: Subtask test [2021-07-01 10:36:40,243] {dagbag.py:487} INFO - Filling up the DagBag from /home/gfm_ops/gfm_dags/data_preparation.py
[2021-07-01 10:36:40,645] {taskinstance.py:1280} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_EMAIL=airflow@example.com
AIRFLOW_CTX_DAG_OWNER=gfm_ops
AIRFLOW_CTX_DAG_ID=GFM-data-preparation
AIRFLOW_CTX_TASK_ID=test
AIRFLOW_CTX_EXECUTION_DATE=2021-07-01T10:36:37.229404+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2021-07-01T10:36:37.229404+00:00
[2021-07-01 10:36:40,666] {docker.py:231} INFO - Starting docker container from image debian:sid-slim
[2021-07-01 10:36:40,678] {taskinstance.py:1481} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/opt/airflow/lib/python3.8/site-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/opt/airflow/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http+docker://localhost/v1.41/containers/create

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/airflow/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1137, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/opt/airflow/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1311, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/opt/airflow/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1341, in _execute_task
    result = task_copy.execute(context=context)
  File "/opt/airflow/lib/python3.8/site-packages/airflow/providers/docker/operators/docker.py", line 319, in execute
    return self._run_image()
  File "/opt/airflow/lib/python3.8/site-packages/airflow/providers/docker/operators/docker.py", line 237, in _run_image
    self.container = self.cli.create_container(
  File "/opt/airflow/lib/python3.8/site-packages/docker/api/container.py", line 430, in create_container
    return self.create_container_from_config(config, name)
  File "/opt/airflow/lib/python3.8/site-packages/docker/api/container.py", line 441, in create_container_from_config
    return self._result(res, True)
  File "/opt/airflow/lib/python3.8/site-packages/docker/api/client.py", line 274, in _result
    self._raise_for_status(response)
  File "/opt/airflow/lib/python3.8/site-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/opt/airflow/lib/python3.8/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 400 Client Error for http+docker://localhost/v1.41/containers/create: Bad Request ("invalid mount config for type "bind": bind source path does not exist: /tmp/airflowtmp6h76o3kr")
[2021-07-01 10:36:40,680] {base_task_runner.py:115} INFO - Job 11: Subtask test Running <TaskInstance: GFM-data-preparation.test 2021-07-01T10:36:37.229404+00:00 [running]> on host 10.10.103.238
[2021-07-01 10:36:40,684] {base_task_runner.py:115} INFO - Job 11: Subtask test Traceback (most recent call last):
[2021-07-01 10:36:40,684] {base_task_runner.py:115} INFO - Job 11: Subtask test   File "/opt/airflow/lib/python3.8/site-packages/docker/api/client.py", line 268, in _raise_for_status
[2021-07-01 10:36:40,684] {base_task_runner.py:115} INFO - Job 11: Subtask test     response.raise_for_status()
[2021-07-01 10:36:40,684] {base_task_runner.py:115} INFO - Job 11: Subtask test   File "/opt/airflow/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status
[2021-07-01 10:36:40,684] {base_task_runner.py:115} INFO - Job 11: Subtask test     raise HTTPError(http_error_msg, response=self)
[2021-07-01 10:36:40,685] {base_task_runner.py:115} INFO - Job 11: Subtask test requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http+docker://localhost/v1.41/containers/create
[2021-07-01 10:36:40,685] {base_task_runner.py:115} INFO - Job 11: Subtask test 
[2021-07-01 10:36:40,685] {base_task_runner.py:115} INFO - Job 11: Subtask test During handling of the above exception, another exception occurred:
[2021-07-01 10:36:40,685] {base_task_runner.py:115} INFO - Job 11: Subtask test 
[2021-07-01 10:36:40,685] {base_task_runner.py:115} INFO - Job 11: Subtask test Traceback (most recent call last):
[2021-07-01 10:36:40,685] {base_task_runner.py:115} INFO - Job 11: Subtask test   File "/opt/airflow/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1137, in _run_raw_task
[2021-07-01 10:36:40,685] {base_task_runner.py:115} INFO - Job 11: Subtask test     self._prepare_and_execute_task_with_callbacks(context, task)
[2021-07-01 10:36:40,685] {base_task_runner.py:115} INFO - Job 11: Subtask test   File "/opt/airflow/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1311, in _prepare_and_execute_task_with_callbacks
[2021-07-01 10:36:40,685] {base_task_runner.py:115} INFO - Job 11: Subtask test     result = self._execute_task(context, task_copy)
[2021-07-01 10:36:40,685] {base_task_runner.py:115} INFO - Job 11: Subtask test   File "/opt/airflow/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1341, in _execute_task
[2021-07-01 10:36:40,685] {base_task_runner.py:115} INFO - Job 11: Subtask test     result = task_copy.execute(context=context)
[2021-07-01 10:36:40,686] {base_task_runner.py:115} INFO - Job 11: Subtask test   File "/opt/airflow/lib/python3.8/site-packages/airflow/providers/docker/operators/docker.py", line 319, in execute
[2021-07-01 10:36:40,686] {base_task_runner.py:115} INFO - Job 11: Subtask test     return self._run_image()
[2021-07-01 10:36:40,686] {base_task_runner.py:115} INFO - Job 11: Subtask test   File "/opt/airflow/lib/python3.8/site-packages/airflow/providers/docker/operators/docker.py", line 237, in _run_image
[2021-07-01 10:36:40,686] {base_task_runner.py:115} INFO - Job 11: Subtask test     self.container = self.cli.create_container(
[2021-07-01 10:36:40,686] {base_task_runner.py:115} INFO - Job 11: Subtask test   File "/opt/airflow/lib/python3.8/site-packages/docker/api/container.py", line 430, in create_container
[2021-07-01 10:36:40,686] {base_task_runner.py:115} INFO - Job 11: Subtask test     return self.create_container_from_config(config, name)
[2021-07-01 10:36:40,686] {base_task_runner.py:115} INFO - Job 11: Subtask test   File "/opt/airflow/lib/python3.8/site-packages/docker/api/container.py", line 441, in create_container_from_config
[2021-07-01 10:36:40,686] {base_task_runner.py:115} INFO - Job 11: Subtask test     return self._result(res, True)
[2021-07-01 10:36:40,686] {base_task_runner.py:115} INFO - Job 11: Subtask test   File "/opt/airflow/lib/python3.8/site-packages/docker/api/client.py", line 274, in _result
[2021-07-01 10:36:40,686] {base_task_runner.py:115} INFO - Job 11: Subtask test     self._raise_for_status(response)
[2021-07-01 10:36:40,686] {base_task_runner.py:115} INFO - Job 11: Subtask test   File "/opt/airflow/lib/python3.8/site-packages/docker/api/client.py", line 270, in _raise_for_status
[2021-07-01 10:36:40,687] {base_task_runner.py:115} INFO - Job 11: Subtask test     raise create_api_error_from_http_exception(e)
[2021-07-01 10:36:40,687] {base_task_runner.py:115} INFO - Job 11: Subtask test   File "/opt/airflow/lib/python3.8/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
[2021-07-01 10:36:40,687] {base_task_runner.py:115} INFO - Job 11: Subtask test     raise cls(e, response=response, explanation=explanation)
[2021-07-01 10:36:40,687] {base_task_runner.py:115} INFO - Job 11: Subtask test docker.errors.APIError: 400 Client Error for http+docker://localhost/v1.41/containers/create: Bad Request ("invalid mount config for type "bind": bind source path does not exist: /tmp/airflowtmp6h76o3kr")
[2021-07-01 10:36:40,687] {base_task_runner.py:115} INFO - Job 11: Subtask test 
[2021-07-01 10:36:40,687] {base_task_runner.py:115} INFO - Job 11: Subtask test During handling of the above exception, another exception occurred:
[2021-07-01 10:36:40,687] {base_task_runner.py:115} INFO - Job 11: Subtask test 
[2021-07-01 10:36:40,687] {base_task_runner.py:115} INFO - Job 11: Subtask test Traceback (most recent call last):
[2021-07-01 10:36:40,687] {base_task_runner.py:115} INFO - Job 11: Subtask test   File "/usr/bin/airflow", line 8, in <module>
[2021-07-01 10:36:40,687] {base_task_runner.py:115} INFO - Job 11: Subtask test     sys.exit(main())
[2021-07-01 10:36:40,687] {base_task_runner.py:115} INFO - Job 11: Subtask test   File "/opt/airflow/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
[2021-07-01 10:36:40,688] {base_task_runner.py:115} INFO - Job 11: Subtask test     args.func(args)
[2021-07-01 10:36:40,688] {base_task_runner.py:115} INFO - Job 11: Subtask test   File "/opt/airflow/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
[2021-07-01 10:36:40,688] {base_task_runner.py:115} INFO - Job 11: Subtask test     return func(*args, **kwargs)
[2021-07-01 10:36:40,688] {base_task_runner.py:115} INFO - Job 11: Subtask test   File "/opt/airflow/lib/python3.8/site-packages/airflow/utils/cli.py", line 91, in wrapper
[2021-07-01 10:36:40,688] {base_task_runner.py:115} INFO - Job 11: Subtask test     return f(*args, **kwargs)
[2021-07-01 10:36:40,688] {base_task_runner.py:115} INFO - Job 11: Subtask test   File "/opt/airflow/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 237, in task_run
[2021-07-01 10:36:40,688] {base_task_runner.py:115} INFO - Job 11: Subtask test     _run_task_by_selected_method(args, dag, ti)
[2021-07-01 10:36:40,688] {base_task_runner.py:115} INFO - Job 11: Subtask test   File "/opt/airflow/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 66, in _run_task_by_selected_method
[2021-07-01 10:36:40,688] {base_task_runner.py:115} INFO - Job 11: Subtask test     _run_raw_task(args, ti)
[2021-07-01 10:36:40,688] {base_task_runner.py:115} INFO - Job 11: Subtask test   File "/opt/airflow/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 148, in _run_raw_task
[2021-07-01 10:36:40,688] {base_task_runner.py:115} INFO - Job 11: Subtask test     ti._run_raw_task(  # pylint: disable=protected-access
[2021-07-01 10:36:40,688] {base_task_runner.py:115} INFO - Job 11: Subtask test   File "/opt/airflow/lib/python3.8/site-packages/airflow/utils/session.py", line 70, in wrapper
[2021-07-01 10:36:40,689] {base_task_runner.py:115} INFO - Job 11: Subtask test     return func(*args, session=session, **kwargs)
[2021-07-01 10:36:40,689] {base_task_runner.py:115} INFO - Job 11: Subtask test   File "/opt/airflow/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1177, in _run_raw_task
[2021-07-01 10:36:40,689] {base_task_runner.py:115} INFO - Job 11: Subtask test     self.handle_failure(e, test_mode, error_file=error_file)
[2021-07-01 10:36:40,689] {base_task_runner.py:115} INFO - Job 11: Subtask test   File "/opt/airflow/lib/python3.8/site-packages/airflow/utils/session.py", line 70, in wrapper
[2021-07-01 10:36:40,689] {base_task_runner.py:115} INFO - Job 11: Subtask test     return func(*args, session=session, **kwargs)
[2021-07-01 10:36:40,689] {base_task_runner.py:115} INFO - Job 11: Subtask test   File "/opt/airflow/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1487, in handle_failure
[2021-07-01 10:36:40,689] {base_task_runner.py:115} INFO - Job 11: Subtask test     set_error_file(error_file, error)
[2021-07-01 10:36:40,689] {base_task_runner.py:115} INFO - Job 11: Subtask test   File "/opt/airflow/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 124, in set_error_file
[2021-07-01 10:36:40,689] {base_task_runner.py:115} INFO - Job 11: Subtask test     with open(error_file, "wb") as fd:
[2021-07-01 10:36:40,689] {base_task_runner.py:115} INFO - Job 11: Subtask test PermissionError: [Errno 13] Permission denied: '/tmp/tmpnzzqxiqz'
[2021-07-01 10:36:40,932] {local_task_job.py:151} INFO - Task exited with return code 1

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:6 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
potiukcommented, Jul 1, 2021

You need to check your docker engine logs. My guess is that your docker engine has the same permission problem with /tmp folder (but this is a wild guess).

0reactions
nikanor97commented, May 10, 2022

SOLVED

I had the same issue and all “recommended” ways of solving the issue here and setting up mount_dir params as descripted here just lead to other errors. The one solution that helped me was wrapping the invocated by docker code with the VPN (actually this hack was taken from another docker-powered DAG that used VPN and worked well).

So the final solution looks like:

#!/bin/bash

connect_to_vpn.sh &
sleep 10
python3 my_func.py
sleep 10
stop_vpn.sh
wait -n
exit $?

To connect to VPN I used openconnect. The took can be installed with apt install and supports anyconnect protocol (it was my crucial requirement).

Read more comments on GitHub >

github_iconTop Results From Across the Web

Unable to run script within Airflow DockerOperator
I am trying to run a simple python script within a docker run command scheduled with Airflow. I have followed the instructions here...
Read more >
[GitHub] [airflow] raphaelauv commented on issue #17037
HTTPError: 400 Client Error: Bad Request for url: http+docker://localhost/v1.41/containers/create During handling of the above exception, ...
Read more >
How to Fix a 400 Bad Request Error (Causes and Fixes) - Kinsta
The 400 Bad Request error indicates that the server cannot or process the request due to a client error. Read about the common...
Read more >
Airflow DockerOperator unable to mount tmp directory correctly
HTTPError: 400 Client Error: Bad Request for url: http+docker://localhost/v1.41/containers/create During handling of the above exception, ...
Read more >
Reverse Proxy and 400 Bad Request error - CI/CD
HI I have 1 Problem with ONE of my docker containers. I can access 3 of 4 with the help of the reverse...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Airflow Helm charts - better integration with Hashicorp Vault

Next page

Queue support for DaskExecutor using Dask Worker Resources