Not able to use Jinja template in PostgresOperator sql parameter as filename

Apache Airflow version

main (development)

What happened

I am trying to pass a sql parameter as a Jinja Template, but not able to. I am a bit new to Airflow, here is the code, I am not sure if the Jinja templating works in PostgresOperator or not. If there is any other way we can pass please let me know, again I am not trying to pass the jinja template inside the SQL query or file, I am passing the name of the SQL file using the Jinja template which it is not able to render.

What you think should happen instead

The Jinja Template should be rendered at sql parameter in PostgresOperator.

How to reproduce

write_to_postgres = PostgresOperator(
        task_id ="write_to_postgres",
        postgres_conn_id="mypostgres",
        #TODO investigate, why this is not working
        sql ="queries{{ execution_date.hour }}{{ execution_date.day }}.sql"
    )

Output of rendered in PostgresOperator

However, the BashOperator is able to render it properly

  delete_queries_file = BashOperator(
        task_id ="delete_queries_file",
        bash_command="cd /opt/airflow/dags && rm queries{{ execution_date.hour }}{{ execution_date.day }}.sql"
    )

Output of the BashOperator with proper rendered of Jinja Template

UPDATE: One thing I noticed weird is when I do not put .sql in the sql parameter, the template gets rendered properly, which seems weird. Here is the screenshot with Jinja Template rendered without the .sql extension in sql parameter, but it works in bash operator as you can seen in the above code which uses .sql at end.

Operating System

Windows 10

Versions of Apache Airflow Providers

apache-airflow-providers-common-sql==1.2.0 apache-airflow-providers-ftp==3.1.0
apache-airflow-providers-http==4.0.0
apache-airflow-providers-imap==3.0.0
apache-airflow-providers-postgres==5.2.2 apache-airflow-providers-sqlite==3.2.1

Deployment

Docker-Compose

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project’s Code of Conduct

[kameshkotwani]

Thank you for considering it, meanwhile I was able to do the required task using vanilla python and PythonOperator, I have attached the code, might help someone with same requirement as mine.

def _write_to_postgres(execution_date):
    hour = execution_date.hour
    day = execution_date.day
    filename = f"queries{hour}{day}.sql"
    import psycopg2
    try:
        conn = psycopg2.connect(database="airflow",user='airflow', password='airflow', host='postgres', port= '5432')
    except:
        raise AirflowFailException
    else:
        cursor = conn.cursor()
        with open(f"/opt/airflow/dags/{filename}","r") as f:
            for statement in f.readlines():
                print(statement)
                cursor.execute(statement)
        conn.commit()
        cursor.close()

[uranusjr]

Unfortunately this is not possible in the current implementation. Maybe we could support it though, it’s not particularly easy to imagine people are actively naming files like Jinja templates that this feature addition would break compatibility (and we can always add a config for this if we are worried).

In the mean time, I’m changing this to a feature request instead.