target-based Jinja rendering of vars in dbt_project.yml

Describe the bug

When compiling / running models from dbt Cloud, the compiler does not correctly evaluate variables defined using jinja in dbt_project.yml and referenced in a sources: block.

Steps To Reproduce

Create in dbt_project.yml:

vars:
  database_prefix: "{% if target.name == 'dev' %}dev_{% else %}{% endif %}"

Create in sources.yml:

sources:
  - name: tpch
    database: "{{ var('database_prefix') }}raw"

Expected behavior

SQL compilation will resolve the variable to either dev_ or an empty string when running either dbt compile or dbt run

Screenshots and log output

On dbt compile:

System information

Which database are you using dbt with?

• postgres
• redshift
• bigquery
• snowflake
• other (specify: ____________)

The output of dbt --version:

0.18.1

The operating system you’re using: dbt Cloud

[jtcohen6]

Thanks for the bug report @matt-winkler! I get that this is unexpected behavior. The truth is that vars in dbt_project.yml aren’t Jinja-rendered today. dbt needs to know the values of vars before it can render any other parts of the project. This leads to weirdness today whereby var() is in the scope of the dbt_project.yml context, but only if set by --vars (docs).

Ultimately, a solution here would be to clearly establish the parsing order of project files:

• env vars, --vars, dbt_version, and builtins are always available
• parse profile --> get target values
• parse vars (with env vars, --vars, and target available)
• parse the rest of the project file

I’m definitely interested in rationalizing when vars are available, and what’s available to them: see https://github.com/fishtown-analytics/dbt/issues/2955. I don’t feel a ton of immediate priority here, though; in the use case you’re describing, there are several other ways to achieve this env-based behavior:

• Your deployment can override {{ var('database_prefix') }} with a (static string) value passed to --vars
• You could control this behavior via {{ env_var() }} instead

If you do want dbt to handle the database prefixing on the basis of target.name alone, it’s not possible to get any DRYer than this:

version: 2

sources:
  - name: tpch
    database: "{{ 'dev_' if target.name == 'dev' else '' }}raw"
    tables:
      - name: something

[github-actions[bot]]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.