BigQuery Service Account JSON Authentication not working

Describe the bug

Trying to authenticate to BigQuery using Service Account JSON Authentication because I’m running dbt from a docker container. Since storing the creds inside the container is not safe, I am passing each field as environment variable and parsing them in profiles.yml.

Steps To Reproduce

In profiles.yml:

      keyfile_json:
        type: "{{ env_var('BIGQUERY_TYPE') }}"
        project_id: "{{ env_var('BIGQUERY_PROJECT_ID') }}"
        private_key_id: "{{ env_var('BIGQUERY_PRIVATE_KEY_ID') }}"
        private_key: "{{ env_var('BIGQUERY_PRIVATE_KEY') }}"
        client_email: "{{ env_var('BIGQUERY_CLIENT_EMAIL') }}"
        client_id: "{{ env_var('BIGQUERY_CLIENT_ID') }}"
        auth_uri: "{{ env_var('BIGQUERY_AUTH_URI') }}"
        token_uri: "{{ env_var('BIGQUERY_TOKEN_URI') }}"
        auth_provider_x509_cert_url: "{{ env_var('BIGQUERY_AUTH_PROVIDER_X509_CERT_URL') }}"
        client_x509_cert_url: "{{ env_var('BIGQUERY_CLIENT_X509_CERT_URL') }}"

However when trying to run anything I get:

Please log into GCP to continue
Encountered an error:
Got a non-zero returncode running: ['gcloud', 'auth', 'application-default', 'login']

Expected behavior

dbt should run normally

System information

Which database are you using dbt with? bigquery

The output of dbt --version:

# dbt --version
installed version: 0.16.0
   latest version: 0.17.0

Your version of dbt is out of date! You can find instructions for upgrading here:
https://docs.getdbt.com/docs/installation

The operating system you’re using: python:3.7.4-slim

The output of python --version: 3.7.4

Additional context

# gcloud --version
Google Cloud SDK 297.0.0
alpha 2020.06.12
beta 2020.06.12
bq 2.0.58
core 2020.06.12
gsutil 4.51
kubectl 2020.06.12

[obiii]

Hi,

I dnt remeber exactly how it got fixed but I was facing problem with the key. When I copy paste the key into github env secrets some of the characters got changes automatically and thus I started copying the key with “ in the start and the end like this “Somekey” and then it worked.

On Tue, 2 Nov 2021 at 07:46, Mat Miadlikowski @.***> wrote:

I am facing the same issue, @obiii https://github.com/obiii did you ever figure it out?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dbt-labs/dbt-core/issues/2553#issuecomment-957142975, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEHRR62RMOGTR4AFMXVGY53UJ6CMBANCNFSM4N766LIQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[darwinyip]

Fixed. It was an issue parsing the private_key