Jinja statements from vars in schema.yml

Describe the feature

Ability to use jinja loops inside schema yaml would really improve how generic things like sources would be. For example: Consider a situation where a package has an input of either a ref or a source, defined based on a var.

It would be nice to be able to accept a list of sources or refs that would that would be unioned together dynamically based on the entries in the var. For refs, it’s completely possible at the moment but for sources, it’s not possible to write jinja statements (such as if / for loops) in a schema.yml file.

ie:

sources:
  - name: my_source
    tables:
      {% for table_name in var('my_list_of_tables') %}
       - name: "{{table_name}}"
      {% endfor %}

Describe alternatives you’ve considered

Obviously a clear alternate is “dont allow this”, which definitely works. Unfortunately, for a source to be referenced, it needs to be defined in a schema yaml so I haven’t thought of any possible alternatives as of right now.

Who will this benefit?

Package developers

Are you interested in contributing this feature?

Happy to help out if I’m able to! I’m already on the hook for another issue that I’ve been dragging my feet on 😅

[dannybluffmsm]

Has jinja implementation been considered anymore in yml files?

I would like to make sure everything is documented correctly. If I have created a loop within my SQL code, then it would be very beneficial to be able to loop within the yml file too.

For example, it would be good to do something like the below:

# dbt_project.yml

var:
    TIME_PERIODS:
    - 7
    - 30
    - 365

# models/my_model.sql

select current_date() as current_date,
      {% for time in var('TIME_PERIODS') %}
          date_sub(current_date(), interval {{time}} day) as date_{{time}}_days_ago,
      {% endfor %}

# models/my_model.yml

version: 2
models:
  - name: my_model
    columns: 
      - name: current_date
        columns: Current date
        {% for time in var('TIME_PERIODS') %} 
      - name: date_{{time}}_days_ago
        columns: Date {{time}} days ago
        {% endfor %}
 

instead of having to create the yml file manually like so:

# models/my_model.yml

version: 2
models:
  - name: my_model
    columns: 
      - name: current_date
        columns: Current date
      - name: date_7_days_ago
        columns: Date 7 days ago
      - name: date_30_days_ago
        columns: Date 30 days ago
      - name: date_365_days_ago
        columns: Date 365 days ago 

Being able to create a loop helps massively if more variables are added (for example TIME_PERIODS in this case) and removes any possibility of undocumented variables.

[msenyonyi]

In fact, the thing you’re after is possible today, with just a few tweaks:

# dbt_project.yml

vars:
  my_list_of_tables:
    - name: tbl_red
    - name: tbl_blue
    - name: tbl_green

@jtcohen6 Is there a way to do this dynamically? If I have a slowly-changing group of tables in a schema? For example, one day I have name: tbl_red, and the next day this table is gone? I would like to cycle through the tables in the schema and add these to the var 😃