Provide jinja template syntax to access connections

Description

Expose the connection into the jinja template context via conn.value.<connectionname>.{host,port,login,password,extra_config,etc}

Today is possible to conveniently access airflow’s variables in jinja templates using {{ var.value.<variable_name> }}.

There is no equivalent (to my knowledge for connections), I understand that most of the time connection are used programmatically in Operators and Hooks source code, but there are use cases where the connection info has to be pass as parameters to the operators and then it becomes cumbersome to do it without jinja template syntax.

I seen workarounds like using user defined macros to provide get_login(my_conn_id), but I’m after a consistent interface for accessing both variables and connections in the same way

Workaround The following user_defined_macro (from my stackoverflow answer) provides the suggested syntax connection.mssql.host where mssql is the connection name:

   class ConnectionGrabber:
       def __getattr__(self, name):
           return  Connection.get_connection_from_secrets(name)
dag = DAG( user_defined_macros={'connection': ConnectionGrabber()}, ...)
 task = BashOperator(task_id='read_connection', bash_command='echo {{connection.mssql.host }}', dag=dag)

This macro can be added to each DAG individually or to all DAGs via an Airflow’s Plugin. What I suggest is to make this macro part of the default.

Use case / motivation

For example, passing credentials to a KubernetesPodOperator via env_vars today has to be done like this:

connection = Connection.get_connection_from_secrets('somecredentials')
k = KubernetesPodOperator(
  task_id='task1', 
  env_vars={'MY_VALUE': '{{ var.value.my_value }}', 'PWD': conn.password,},
  )

where I would prefer to use consistent syntax for both variables and connections like this:

# not needed anymore: connection = Connection.get_connection_from_secrets('somecredentials')
k = KubernetesPodOperator(
  task_id='task1', 
  env_vars={'MY_VALUE': '{{ var.value.my_value }}', 'PWD':  '{{ conn.somecredentials.password }}',},
  )

The same applies to BashOperator where I sometimes feel the need to pass connection information to the templated script.

Are you willing to submit a PR?

yes, I can write the PR.

Related Issues

[ecerulm]

I asked in the dev mailing list back in 2021-03-21 and the answer was that without the ability to mask passwords in logs and rendered templates it was bad from the security standpoint so I left it there.

It seems that #8421 and #9638 are fixed now. So I will write a PR now.

[turbaszek]

I agree which @mik-laj that top level query is bad idea.

As you said getting connections already can be done by using custom macro. So I’m in favour of implementing standard interface.

What I can suggest is to bring this issue to dev list so we may discuss potential security issues. Having a proof of concept would be nice as it may speed up understanding how many changes we need.