question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[AIP-31] Add lazy way to get Airflow context from a wrapped function

See original GitHub issue

Description

Currently the only way to access the context object from a PythonOperator wrapped function is by setting provide_context=True in the Operator. That loads it into the kwargs. This makes the signature of the function become difficult to argue about.

Adding a way to lazy get the function when executing the function may be better. Similar to Flask flask.request that get populated when the function is executed.

Use case / motivation

Use the Airflow context in arbitrary function while keeping the signature of the function stable and easy to reason about.

An idea of implementation would be:

from airflow.task import context

@task 
def context_task(...)
    return context['run_date']

This can be populated by enabling a context manager on the PythonOperator.execute function that populates the field (similar to how DagContext works) and removes it afterwards.

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:9 (9 by maintainers)

github_iconTop GitHub Comments

1reaction
casassgcommented, Apr 2, 2020

I see, I still think we should implement this lazy resolved context. Using the current approach from master means that signatures can mismatch op_args/op_kwargs and would make it difficult to reason about the function signature:

@task 
def simple_task(text:str, execution_date: datetime):
  print(text)
  print(execution_date)

simple_task("hello")

As you can see this will execute correctly even though the function signature has a an extra argument. This leads to confusion in my personal opinion hence my proposed solution.

0reactions
casassgcommented, Jun 10, 2020

Any updates on this?

Read more comments on GitHub >

github_iconTop Results From Across the Web

Airflow functional DAG definition - AIP-31 - YouTube
Gerard Casas SaezAirflow does not currently have an explicit way to declare messages passed between tasks in a DAG. XCom are available but ......
Read more >
Creating a custom Operator - Apache Airflow
The method contains the Airflow context as a parameter that can be used to read config values. Note. When implementing custom operators, do...
Read more >
Concepts — Airflow Documentation
In addition, you can wrap functions as tasks using the task decorator. ... In this way you can gain access to context dictionary...
Read more >
AIP-31: "TaskFlow API" for clearer/simpler DAG definition
This AIP focuses on proposing a Functional API that would allow to explicitly declare message passing while implicitly declaring task dependencies. At the...
Read more >
airflow.models.baseoperator — Airflow Documentation
Abstract base class that defines how we get an operator link. Functions¶. parse_retries (retries).
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found