Can't access BigQuery tables whose original sources are spreadsheets on Google Drive with the oauth method

Describe the bug

It is impossible to access BigQuery tables whose original sources are spreadsheets on Google Drive with the oauth method, because we don’t pass any scopes parameters to the method to create BigQuery credentials. We need a scope of https://www.googleapis.com/auth/drive to access such tables. Moreover, there is no way to grant scopes, not permissions, to a google cloud authentication. We can only pass the scope information when creating a credentials.

• https://github.com/fishtown-analytics/dbt/blob/v0.19.0/plugins/bigquery/dbt/adapters/bigquery/connections.py#L58

Steps To Reproduce

1. Create a BigQuery table whose data source is a spreadsheet.
2. Set a profile with the oauth method for BigQuery.
3. Create a dbt model which refers to the BigQuery table with the profile.

Expected behavior

Even when we use the oauth method of BigQuery profile, we should be able to access BigQuery tables whose original data sources are spreadsheets on Google Drive with the scope of https://www.googleapis.com/auth/drive.

Screenshots and log output

$ dbt run
...
Database Error in model my_model (models/my_model.sql)
  Access Denied: BigQuery BigQuery: Permission denied while getting Drive credentials.
  compiled SQL at target/run/my_model.sql

System information

Which database are you using dbt with?

• postgres
• redshift
• bigquery
• snowflake
• other (specify: ____________)

The output of dbt --version:

installed version: 0.19.0
   latest version: 0.19.0

Up to date!

Plugins:
  - bigquery: 0.19.0

The operating system you’re using: Mac OS 10.15.7 (19H114)

The output of python --version: Python 3.8.5

[walshie4]

Okay I’m back, and I have good news. After digging through here and trying out some of the changes I think I understand how this happened and how we can resolve the issue.

To start the issue was first introduced in PR #2908 where this line which passed scopes when using OAuth was replaced with this line which relies on the defaults configured with gcloud.

This is why even after this change if you authenticate with the scopes added (using gcloud auth application-default login --scopes=https://www.googleapis.com/auth/bigquery, https://www.googleapis.com/auth/drive.readonly you will not see the failure since your default credentials have the scopes added.

However, if you have a setup like we do over at Honey 😉 you will see the problem because we use a service account in an environment with no default credentials available and authenticate by activating the service account with gcloud auth activate-service-account which does not allow for passing scopes, but will cause the dbt auth code to still go down that (OAuth) path. This is a fairly standard auth flow for GCP (and is a common approach for production authentication with service accounts)

Looking back now this is actually not a mentioned path on the BQ profile setup page (can’t remember if it was back a ~year ago when we set this up) which may be why this path got hit with the bug.

Tested our setup with #3041 and it does resolve the issue (manually adding the 4 lines of changes causes a previous failure to then work in the same environment back-to-back).

I think we should merge that change, and consider updating the docs to cover the activate-service-account authentication path.

Let me know if you have any questions. I also support making these optional in some form for those who don’t need them.

[heisencoder]

Okay, I think I found the fix. It’s necessary to run this command to add the Drive scope to the application default credentials:

gcloud auth application-default login --scopes=https://www.googleapis.com/auth/drive,https://www.googleapis.com/auth/bigquery

(Note that you also have to add the bigquery scope and anything else needed by dbt).