question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

@task decorator does not handle "templates_exts" argument correctly (files are not rendered)

See original GitHub issue

Apache Airflow version

2.2.2

What happened

When I use Taskflow, and especially the @task decorator to create a task, the template files are not correctly rendered : I still have the filename inside templates_dict instead of the templated file content, even though I supplied a valid templates_exts argument (at least one which works with PythonOperator).

I do not encounter the same issue using the old PythonOperator syntax.

What you expected to happen

I expect that the behavior of the @task decorator would be the same as the PythonOperator class, or at least have a workaround explained in the documentation such as other arguments I would need to provide.

How to reproduce

Sample code below :

from airflow.decorators import task

@task(templates_dict={"query": "sql/sample.sql"}, templates_exts=[".sql"])
def aggregate_logs(**kwargs):
    logging.info("query: %s", str(kwargs["templates_dict"]["query"]))

Which returns INFO - query: sql/sample.sql.

However, if I use a PythonOperator with the old syntax it works :

from airflow.operators.python import PythonOperator

def aggregate_logs(**kwargs):
    logging.info("query: %s", str(kwargs["templates_dict"]["query"]))

aggregate_task = PythonOperator(
    python_callable=aggregate_logs,
    templates_dict={"query": "sql/sample.sql"},
    templates_exts=[".sql"],
)

I get the entire templated file inside kwargs["templates_dict"]["query"], not just its name.

Operating System

AWS MWAA

Versions of Apache Airflow Providers

Not sure if relevant here since I only use Airflow packages in the example.

Deployment

MWAA

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:7 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
Bowrnacommented, Sep 12, 2022

I could try fixing this issue if @xingularity and @esthomw do not want to take over.

0reactions
esthomwcommented, Sep 13, 2022

@potiuk I would love to address that issue, however, I am not sure if I currently have enough time. In that case please go ahead @Bowrna 😃 I will try to help with reviewing and testing

Read more comments on GitHub >

github_iconTop Results From Across the Web

Airflow using template files for PythonOperator - Stack Overflow
I am not sure how the templates_exts and templates_dict parameters would correctly interact to pick up a file. In my dags folder I've...
Read more >
Use Airflow templates | Astronomer Documentation
Use Airflow templates. Templating allows you to pass dynamic information into task instances at runtime. For example, you can run the following command...
Read more >
Concepts — Airflow Documentation
Airflow task decorator converts any Python function to an Airflow operator. The decorated function can be called once to set the arguments and...
Read more >
Creating a custom Operator - Apache Airflow Documentation
Use @apply_defaults decorator function to fill unspecified arguments with default_args . You can specify the default_args in the dag file.
Read more >
Upgrading from 1.x — Fabric documentation
The CLI task-oriented workflow remains a primary design goal, but the library use case is no longer a second-class citizen; instead, the library...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found