BigQuery Service Account JSON Authentication not working with env_var

Describe the bug

Same that https://github.com/fishtown-analytics/dbt/issues/2553 but

If I just put the value:

private_key: XXX

it works but using env_var it does not.

Steps To Reproduce

Same that https://github.com/fishtown-analytics/dbt/issues/2553

private_key: “{{ env_var(‘BQ_PRIVATE_KEY’) }}” or private_key: “{{env_var(‘BQ_PRIVATE_KEY’)}}”

Expected behavior

dbt should run normally

Screenshots and log output

Could not deserialize key data.

System information

Which database are you using dbt with?

• postgres
• redshift
• bigquery
• snowflake
• other (specify: ____________)

The output of dbt --version:

installed version: 0.18.1

The operating system you’re using: Debian GNU/Linux 10 (buster) --> docker image fishtownanalytics/dbt:0.18.1

The output of python --version: Python 3.8.3

[jtcohen6]

As in #2553 and #2989, this is quite difficult to debug at a distance, but here’s my best guess: The Could not deserialize key data error indicates that some character—often a newline—is added, missing, or in the wrong place.

Private keys contain many newlines, and they’re quite sensitive about them. In JSON, new lines in strings are represented via \n. When you download a JSON-formatted private keyfile from BigQuery, the private_key value will therefore contain \n.

However, when you’re passing the private_key value as a standalone string (not within a larger JSON blob), you need to replace all the instances of \n with a true newline. When setting the environment variable, you want to end up with something like:

export BQ_PRIVATE_KEY="-----BEGIN PRIVATE KEY-----
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxx
-----END PRIVATE KEY-----
"

I’m not positive why var is working, while env_var is not working—I wasn’t able to replicate this!—but it’s possible that printing the env var out to CLI, then back in as an argument to --vars, replaces \n with newlines because it’s a bash string. I don’t think this reflects a functional difference between how dbt handles {{ var() }} vs. {{ env_var() }}.

If you’re still having trouble, or if you don’t care to manually edit the private key, you could also try passing the entire keyfile contents (as a JSON blob) into a single environment variable:

type: bigquery
method: service-account-json
...
keyfile_json: "{{ env_var('BQ_KEYFILE_JSON') | as_native }}"

[venosov]

Thanks, there must be some difference between env_var and var, as var works like a charm with the same complex key.