[BigQuery] Raw token authentication method

Describe the feature

The existing auth methods for BigQuery provide a great experience for interactive users, allowing them to transparently provision access tokens in a variety of ways. However, as an automation implementer on a team trying to programmatically wield dbt on behalf of customers, I’d like to be able to bypass these niceties and just inject an externally provisioned token for access.

In our typical non-dbt scenarios, we create service accounts for customers, have them grant specific limited permissions to those accounts for our service’s operations, then use our master account to issue scoped tokens for those accounts/operations when they need to execute. When running dbt operations, I’d like to avoid writing our master account credentials file to disk (as required by the service-account/service-account-json methods) to protect against reflected file attacks and/or potential vulnerabilities in dbt itself, which could potentially exfiltrate these creds. While I don’t see these as particularly likely scenarios, I’d love to have the ability to directly control the blast radius via smaller scoped credentials.

I propose a new BigQuery auth method named ‘service-token’ and an additional field named service_token to provide its value, e.g.:

my-bigquery-db:
  target: dev
  outputs:
    dev:
      type: bigquery

      # something like this?
      method: service-token
      service_token: 'ya29.c.KqwC3gfRlXO...'

      project: [GCP project id]
      dataset: [the name of your dbt dataset]
      threads: [1 or more]
      timeout_seconds: 300
      priority: interactive
      retries: 1

Describe alternatives you’ve considered

With the addition of the impersonate_service_account field added in 0.18.0, I can connect as delegated services, but still need to provide the full master service keyfile rather than a more limited credential.

Additional context

This is specific to BigQuery database connections and automation use-cases. I think it’s fair to say that standard interactive users would almost never want to use this authentication mode directly.

Who will this benefit?

• Security-minded SaaS automators who are targeting multi-warehouse and connecting to BQ on behalf of customers
• IT/Dataeng staff using automation to provision limited credentials as part of employee access control

Are you interested in contributing this feature?

Yes, happy to contribute to making this a reality. I have a simple (if slightly hacky) proof-of-concept that I can polish into a real PR if this seems interesting.

[davehughes]

Yep, just pulled that down and confirmed that it works for me with the following settings:

type: bigquery
method: oauth-secrets
token: <my generated token>
# refresh_token/client_id/client_secret/token_uri set to None

Passing a bad or expired token errors with the Unable to generate access token message, which is reasonable for folks using this particular approach.

Thanks @drewbanin @jtcohen6 for the nice solution. I’ll look forward to this landing. 😎

[uaroracca]

Hello, the discussion above is super helpful. I just had another related issue if someone could help with this.

Issue:

profiles.yml:

same config was previously working but after a few months it’s causing this issue. Any help is appreciated.

Thanks.