Allow custom variables inside profiles.yml

Describe the feature

I would like to be able to create and set custom variables inside ~/.dbt/profiles.yml file and then use these variables within models.

Describe alternatives you’ve considered

• Setting environment variables. However this would require further setup instructions to ask people to add variables to their .profile, .bash_rc, .zshrc etc. which requires additional command line knowledge. It would also be harder to phase more variables in and deprecate them.
• Creating a macro that can parse a yaml file (i.e. with open('~/.dbt/profiles.yml') ... ) and setting variables from there. However this is not currently possibly.

Additional context

The goal is to make the initial setup process for installing dbt as simple as possible. A lot of the users will not have command line experience, so it’s a big step to get them to update their ~/.dbt/profiles.yml file. Therefore asking them to also maintain their own environment variables is not ideal and we would be able to provision these variables within our GitHub repo (and deploy efficiently).

This would also allow greater flexibility for what we can conditionally do inside dbt_project.yml and each of the individual models.

Who will this benefit?

The driver behind this is for everyone to set a DBT_USER_ALIAS variable that be used as a prefix to table names (inside generate_alias_name) and therefore prevent conflicts from occurring.

It will benefit those without that much command line experience, and subsequently administrators who spend time supporting them.

[cwelton]

Adding my voice to the others asking for this. Being able to set variables as part of a profile is a much cleaner solution than putting that configuration into conditional expressions or requiring additional setting of environment variables.

The choice to not implement this as a capability feels like the wrong decision.

[jtcohen6]

@vembloud How many different environments are you running against? Have you considered conditional logic like:

sources:
  - name: extract
    database: "{{ 'extract_prod' if target.name == 'prod' else 'extract_dev' }}"
    schema: extract
    tables: ...

In general, we find a pretty smooth handoff point from “I’m running with a few environments and can write target-based conditional logic” to “I have a complex deployment with several environments, codified in an orchestration tool that can also reliably set environment variables.”