Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

execution_timeout doesn't work with PythonVirtualenvOperator

See original GitHub issue

Apache Airflow version

2.2.2

What happened

When using execute_timeout with PythonVirtualenvOperator, the AirflowTaskTimeout exception is raised but it does not fail the task.

What you think should happen instead

When the timeout hits, the task should fail like it does for all the other operators.

How to reproduce

Here is my test DAG

from datetime import datetime, timedelta

from airflow.models import DAG
from airflow.operators.python import PythonOperator, PythonVirtualenvOperator

args = {
    'owner': 'test',
    'start_date': datetime(2022, 5, 24, 0, 0)
}

dag = DAG(
    dag_id='test_python_venv',
    default_args=args,
    schedule_interval=None,
    catchup=False,
    dagrun_timeout=timedelta(minutes=10)
)

def my_callable():
    import time
    print("Sleeping...")
    time.sleep(1000000)

PythonVirtualenvOperator(
    task_id="python_venv",
    dag=dag,
    python_callable=my_callable,
    execution_timeout=timedelta(minutes=1)
)

PythonOperator(
    task_id="python_no_venv",
    dag=dag,
    python_callable=my_callable,
    execution_timeout=timedelta(minutes=1)
)

The Python operator task will timeout like expected, but not the PythonVirtualenvOperator.

While running debug logs, here is the output of the PythonVirtualenvOperator task:

[2022-05-24, 12:42:56 UTC] {taskinstance.py:1427} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=test
AIRFLOW_CTX_DAG_ID=test_python_venv
AIRFLOW_CTX_TASK_ID=python_venv
AIRFLOW_CTX_EXECUTION_DATE=2022-05-24T16:41:34.776017+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2022-05-24T16:41:34.776017+00:00
[2022-05-24, 12:42:56 UTC] {__init__.py:146} DEBUG - Preparing lineage inlets and outlets
[2022-05-24, 12:42:56 UTC] {__init__.py:190} DEBUG - inlets: [], outlets: []
[2022-05-24, 12:42:56 UTC] {process_utils.py:135} INFO - Executing cmd: /usr/bin/python3 -m virtualenv /tmp/venvvwhxg2m8 --system-site-packages
[2022-05-24, 12:42:56 UTC] {process_utils.py:139} INFO - Output:
[2022-05-24, 12:42:59 UTC] {process_utils.py:143} INFO - created virtual environment CPython3.8.10.final.0-64 in 2710ms
[2022-05-24, 12:42:59 UTC] {process_utils.py:143} INFO -   creator CPython3Posix(dest=/tmp/venvvwhxg2m8, clear=False, global=True)
[2022-05-24, 12:42:59 UTC] {process_utils.py:143} INFO -   seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, pkg_resources=latest, via=copy, app_data_dir=/home/sm/.local/share/virtualenv/seed-app-data/v1.0.1.debian.1)
[2022-05-24, 12:42:59 UTC] {process_utils.py:143} INFO -   activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
[2022-05-24, 12:42:59 UTC] {process_utils.py:135} INFO - Executing cmd: /tmp/venvvwhxg2m8/bin/python /tmp/venvvwhxg2m8/script.py /tmp/venvvwhxg2m8/script.in /tmp/venvvwhxg2m8/script.out /tmp/venvvwhxg2m8/string_args.txt
[2022-05-24, 12:42:59 UTC] {process_utils.py:139} INFO - Output:
[2022-05-24, 12:43:00 UTC] {process_utils.py:143} INFO - [2022-05-24, 12:43:00 UTC] {settings.py:210} DEBUG - Setting up DB connection pool (PID 158989)
[2022-05-24, 12:43:00 UTC] {process_utils.py:143} INFO - [2022-05-24, 12:43:00 UTC] {plugins_manager.py:287} DEBUG - Loading plugins
[2022-05-24, 12:43:00 UTC] {process_utils.py:143} INFO - [2022-05-24, 12:43:00 UTC] {plugins_manager.py:231} DEBUG - Loading plugins from directory: /home/sm/airflow/plugins
[2022-05-24, 12:43:00 UTC] {process_utils.py:143} INFO - [2022-05-24, 12:43:00 UTC] {plugins_manager.py:211} DEBUG - Loading plugins from entrypoints
[2022-05-24, 12:43:00 UTC] {process_utils.py:143} INFO - [2022-05-24, 12:43:00 UTC] {plugins_manager.py:445} DEBUG - Integrate DAG plugins
[2022-05-24, 12:43:01 UTC] {taskinstance.py:720} DEBUG - Refreshing TaskInstance <TaskInstance: test_python_venv.python_venv manual__2022-05-24T16:41:34.776017+00:00 [running]> from DB
[2022-05-24, 12:43:01 UTC] {taskinstance.py:761} DEBUG - Refreshed TaskInstance <TaskInstance: test_python_venv.python_venv manual__2022-05-24T16:41:34.776017+00:00 [running]>
[2022-05-24, 12:43:01 UTC] {base_job.py:227} DEBUG - [heartbeat]
[2022-05-24, 12:43:06 UTC] {taskinstance.py:720} DEBUG - Refreshing TaskInstance <TaskInstance: test_python_venv.python_venv manual__2022-05-24T16:41:34.776017+00:00 [running]> from DB
[2022-05-24, 12:43:06 UTC] {taskinstance.py:761} DEBUG - Refreshed TaskInstance <TaskInstance: test_python_venv.python_venv manual__2022-05-24T16:41:34.776017+00:00 [running]>
[2022-05-24, 12:43:06 UTC] {base_job.py:227} DEBUG - [heartbeat]
[2022-05-24, 12:43:11 UTC] {taskinstance.py:720} DEBUG - Refreshing TaskInstance <TaskInstance: test_python_venv.python_venv manual__2022-05-24T16:41:34.776017+00:00 [running]> from DB
[2022-05-24, 12:43:11 UTC] {taskinstance.py:761} DEBUG - Refreshed TaskInstance <TaskInstance: test_python_venv.python_venv manual__2022-05-24T16:41:34.776017+00:00 [running]>
[2022-05-24, 12:43:11 UTC] {base_job.py:227} DEBUG - [heartbeat]
[2022-05-24, 12:43:16 UTC] {taskinstance.py:720} DEBUG - Refreshing TaskInstance <TaskInstance: test_python_venv.python_venv manual__2022-05-24T16:41:34.776017+00:00 [running]> from DB
[2022-05-24, 12:43:16 UTC] {taskinstance.py:761} DEBUG - Refreshed TaskInstance <TaskInstance: test_python_venv.python_venv manual__2022-05-24T16:41:34.776017+00:00 [running]>
[2022-05-24, 12:43:16 UTC] {base_job.py:227} DEBUG - [heartbeat]
[2022-05-24, 12:43:21 UTC] {taskinstance.py:720} DEBUG - Refreshing TaskInstance <TaskInstance: test_python_venv.python_venv manual__2022-05-24T16:41:34.776017+00:00 [running]> from DB
[2022-05-24, 12:43:21 UTC] {taskinstance.py:761} DEBUG - Refreshed TaskInstance <TaskInstance: test_python_venv.python_venv manual__2022-05-24T16:41:34.776017+00:00 [running]>
[2022-05-24, 12:43:21 UTC] {base_job.py:227} DEBUG - [heartbeat]
[2022-05-24, 12:43:26 UTC] {taskinstance.py:720} DEBUG - Refreshing TaskInstance <TaskInstance: test_python_venv.python_venv manual__2022-05-24T16:41:34.776017+00:00 [running]> from DB
[2022-05-24, 12:43:26 UTC] {taskinstance.py:761} DEBUG - Refreshed TaskInstance <TaskInstance: test_python_venv.python_venv manual__2022-05-24T16:41:34.776017+00:00 [running]>
[2022-05-24, 12:43:26 UTC] {base_job.py:227} DEBUG - [heartbeat]
[2022-05-24, 12:43:31 UTC] {taskinstance.py:720} DEBUG - Refreshing TaskInstance <TaskInstance: test_python_venv.python_venv manual__2022-05-24T16:41:34.776017+00:00 [running]> from DB
[2022-05-24, 12:43:31 UTC] {taskinstance.py:761} DEBUG - Refreshed TaskInstance <TaskInstance: test_python_venv.python_venv manual__2022-05-24T16:41:34.776017+00:00 [running]>
[2022-05-24, 12:43:31 UTC] {base_job.py:227} DEBUG - [heartbeat]
[2022-05-24, 12:43:36 UTC] {taskinstance.py:720} DEBUG - Refreshing TaskInstance <TaskInstance: test_python_venv.python_venv manual__2022-05-24T16:41:34.776017+00:00 [running]> from DB
[2022-05-24, 12:43:36 UTC] {taskinstance.py:761} DEBUG - Refreshed TaskInstance <TaskInstance: test_python_venv.python_venv manual__2022-05-24T16:41:34.776017+00:00 [running]>
[2022-05-24, 12:43:36 UTC] {base_job.py:227} DEBUG - [heartbeat]
[2022-05-24, 12:43:41 UTC] {taskinstance.py:720} DEBUG - Refreshing TaskInstance <TaskInstance: test_python_venv.python_venv manual__2022-05-24T16:41:34.776017+00:00 [running]> from DB
[2022-05-24, 12:43:41 UTC] {taskinstance.py:761} DEBUG - Refreshed TaskInstance <TaskInstance: test_python_venv.python_venv manual__2022-05-24T16:41:34.776017+00:00 [running]>
[2022-05-24, 12:43:41 UTC] {base_job.py:227} DEBUG - [heartbeat]
[2022-05-24, 12:43:46 UTC] {taskinstance.py:720} DEBUG - Refreshing TaskInstance <TaskInstance: test_python_venv.python_venv manual__2022-05-24T16:41:34.776017+00:00 [running]> from DB
[2022-05-24, 12:43:46 UTC] {taskinstance.py:761} DEBUG - Refreshed TaskInstance <TaskInstance: test_python_venv.python_venv manual__2022-05-24T16:41:34.776017+00:00 [running]>
[2022-05-24, 12:43:46 UTC] {base_job.py:227} DEBUG - [heartbeat]
[2022-05-24, 12:43:51 UTC] {taskinstance.py:720} DEBUG - Refreshing TaskInstance <TaskInstance: test_python_venv.python_venv manual__2022-05-24T16:41:34.776017+00:00 [running]> from DB
[2022-05-24, 12:43:51 UTC] {taskinstance.py:761} DEBUG - Refreshed TaskInstance <TaskInstance: test_python_venv.python_venv manual__2022-05-24T16:41:34.776017+00:00 [running]>
[2022-05-24, 12:43:51 UTC] {base_job.py:227} DEBUG - [heartbeat]
[2022-05-24, 12:43:56 UTC] {timeout.py:36} ERROR - Process timed out, PID: 158977
[2022-05-24, 12:43:56 UTC] {taskinstance.py:720} DEBUG - Refreshing TaskInstance <TaskInstance: test_python_venv.python_venv manual__2022-05-24T16:41:34.776017+00:00 [running]> from DB
[2022-05-24, 12:43:56 UTC] {taskinstance.py:761} DEBUG - Refreshed TaskInstance <TaskInstance: test_python_venv.python_venv manual__2022-05-24T16:41:34.776017+00:00 [running]>
[2022-05-24, 12:43:56 UTC] {base_job.py:227} DEBUG - [heartbeat]

Operating System

NAME=“Ubuntu” VERSION=“20.04.2 LTS (Focal Fossa)”

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:7 (6 by maintainers)

github_iconTop GitHub Comments

1reaction
potiukcommented, May 31, 2022

I think the best place to describe it is “timeout” parameter docsctring in BaseOperator.

1reaction
potiukcommented, May 31, 2022

This is correct. Airflow does not - on its own propagate - SIGALRM (this is how timeout is implemented) to subprocesses. And your subprocess would have to handle the SIGALRM in this case anyway. Airflow timeout only works if you run a code in the task process but you do more.

Peopen’s contxt manager is documented here: https://docs.python.org/3/library/subprocess.html#subprocess.Popen:

Popen objects are supported as context managers via the [with (https://docs.python.org/3/reference/compound_stmts.html#with) statement: on exit, standard file descriptors are closed, and the process is waited for.

You are starting subprocess and waiting for it, the only way to stop the waiting is to kill the process, but airflow has no notion of the sub-process you started so it cannot do much there (short of killing all the process group forcefully which it should not do), So you should manage both timeout and killing the subprocess on your own.

The easy way to do it it s to start your subprocess without context manager (i.e. with) and explicitly wait for it with timeout https://docs.python.org/3/library/subprocess.html#popen-objects and terminate the process in the way you think is appropriate after the timeout expires. Airflow simply cannot do it for you because it does not know how many and what kind of subprocesses you started. Only your code knows it and only your code can terminate those sub-processes in a gentle (or not if you choose so) way.

Read more comments on GitHub >

github_iconTop Results From Across the Web

XCOM's don't work with PythonVirtualenvOperator airflow 1.10.6
I'm using the PythonVirtualenvOperator to have my tasks individualized and have different requeremets/versions in each task. Still not working ...
Read more >
Cloud Composer release notes | Google Cloud
These release notes apply to the Cloud Composer service. You can periodically check this page for announcements about new or updated features, bug...
Read more >
PythonVirtualenvOperator givig PicklingError - Google Groups
I am trying to run a PythonVirtualenvOperator within a dag in a gcloud composer environment but am getting an error while the virtual...
Read more >
Creating a custom plugin for Apache Airflow ...
This plugin will patch the built-in PythonVirtualenvOperator during that startup process to make it compatible with Amazon MWAA. The following steps show ...
Read more >
How did I resolved pip package dependency issue in Apache ...
Talks about the cyclic dependency issue with pip packages, how to resolve it using PythonVirtualenvOperator. Airflow and pip packages dependency ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Airflow 2.3.1 - gunicorn keeps removing and adding Permission menu access on Permissions to role Admin

Next page

Airflow DAGs not refreshed with pullPolicy set to Always with the same container tag