DockerOperator: OCI Runtime create failed: exec: "PYTHONPATH=.": executable file not found in $PATH: unknown

Apache Airflow version: 2.0.0

• OS (e.g. from /etc/os-release): PRETTY_NAME=“Debian GNU/Linux 10 (buster)” NAME=“Debian GNU/Linux” VERSION_ID=“10” VERSION=“10 (buster)” VERSION_CODENAME=buster ID=debian HOME_URL=“https://www.debian.org/” SUPPORT_URL=“https://www.debian.org/support” BUG_REPORT_URL=“https://bugs.debian.org/”

What happened:

Hello, I am trying to run a python command using DockerOperator. In the command field, I am passing PYTHONPATH=. as an argument. The command which is being used is attached below.

PYTHONPATH=. python3 scripts/xyz.py

When I pass the PYTHONPATH command here, I am getting the error:

docker.errors.APIError: 500 Server Error: Internal Server Error ("OCI runtime create failed: container_linux.go:370: starting container process caused: exec: "PYTHONPATH=.": executable file not found in $PATH: unknown")

Here’s the complete traceback from the logs:

Logs from DockerOperator

[2021-03-02 17:11:49,329] {taskinstance.py:1396} ERROR - 500 Server Error: Internal Server Error ("OCI runtime create failed: container_linux.go:370: starting container process caused: exec: "PYTHONPATH=.": executable file not found in $PATH: unknown") Traceback (most recent call last): File "/home/airflow/.local/lib/python3.6/site-packages/docker/api/client.py", line 261, in _raise_for_status response.raise_for_status() File "/home/airflow/.local/lib/python3.6/site-packages/requests/models.py", line 941, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: xxxxxxx

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File “/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py”, line 1086, in _run_raw_task self._prepare_and_execute_task_with_callbacks(context, task) File “/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py”, line 1260, in _prepare_and_execute_task_with_callbacks result = self._execute_task(context, task_copy) File “/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py”, line 1300, in _execute_task result = task_copy.execute(context=context) File “/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/docker/operators/docker.py”, line 305, in execute return self._run_image() File “/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/docker/operators/docker.py”, line 255, in _run_image self.cli.start(self.container[‘Id’]) File “/home/airflow/.local/lib/python3.6/site-packages/docker/utils/decorators.py”, line 19, in wrapped return f(self, resource_id, *args, **kwargs) File “/home/airflow/.local/lib/python3.6/site-packages/docker/api/container.py”, line 1091, in start self._raise_for_status(res) File “/home/airflow/.local/lib/python3.6/site-packages/docker/api/client.py”, line 263, in _raise_for_status raise create_api_error_from_http_exception(e) File “/home/airflow/.local/lib/python3.6/site-packages/docker/errors.py”, line 31, in create_api_error_from_http_exception raise cls(e, response=response, explanation=explanation) docker.errors.APIError: 500 Server Error: Internal Server Error (“OCI runtime create failed: container_linux.go:370: starting container process caused: exec: “PYTHONPATH=.”: executable file not found in $PATH: unknown”)

I tried passing the complete path of the python.exe file as well like:

/bin/usr/python3 scripts/xyz.py

With this as well, it threw the same error.

I also tried executing directly on the target docker daemon with command like:

docker -H tcp://0.0.0.0:2375 run -v /xyz:/xyz -it <docker_image_name> bash -c "PYTHONPATH=. python3 scripts/xyz.py"

The above command works as expected, but through the DockerOperator it does not look like it is working.

Anything else we need to know: The problem occurs whenever I add something before the python3 command.

[xinbinhuang]

Does below work? The command should be the same as what you pass in ... -v /xyz:/xyz -it <docker_image_name> <command> using normal docker command

t = DockerOperator(
                ....,
                command='bash -c "PYTHONPATH=. python3 scripts/xyz.py"',
                mounts=[...],
                docker_url="tcp://0.0.0.0:2375",
                ...
        )

[kaustubhharapanahalli]

This still doesn’t work when passing the PYTHONPATH=. in command. It has to passed as environment variable - solution if anyone else faces this issue.