[AIP-31] Add lazy way to get Airflow context from a wrapped function
See original GitHub issueDescription
Currently the only way to access the context object from a PythonOperator wrapped function is by setting provide_context=True
in the Operator. That loads it into the kwargs. This makes the signature of the function become difficult to argue about.
Adding a way to lazy get the function when executing the function may be better. Similar to Flask flask.request
that get populated when the function is executed.
Use case / motivation
Use the Airflow context in arbitrary function while keeping the signature of the function stable and easy to reason about.
An idea of implementation would be:
from airflow.task import context
@task
def context_task(...)
return context['run_date']
This can be populated by enabling a context manager on the PythonOperator.execute
function that populates the field (similar to how DagContext
works) and removes it afterwards.
Issue Analytics
- State:
- Created 3 years ago
- Comments:9 (9 by maintainers)
Top GitHub Comments
I see, I still think we should implement this lazy resolved context. Using the current approach from master means that signatures can mismatch op_args/op_kwargs and would make it difficult to reason about the function signature:
As you can see this will execute correctly even though the function signature has a an extra argument. This leads to confusion in my personal opinion hence my proposed solution.
Any updates on this?