Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

push: gdrive_service_account_json_file_path must be set for GDRIVE_CREDENTIALS_DATA to work

See original GitHub issue

Description

The GDRIVE_CREDENTIALS_DATA environment variable won’t be honored unless the gdrive_service_account_json_file_path configuration option is set beforehand. In most of the habitual use cases, this requirement doesn’t make too much sense from the user standpoint.

Is this the recommended way of passing Google Drive (Service Account) credentials through an environment variable?

Additionally, after reading the pertaining documentation, I don’t understand why I can’t use the GDRIVE_CREDENTIALS_DATA environment variable to pass the DVC-specific .dvc/tmp/gdrive-user-credentials.json file contents, instead of having to pass the raw service account file contents.

How to reproduce

The following GitHub Actions workflow provides a self-contained example to reproduce this issue. Uncommenting the only line prefixed with a hash # will make it work as expected.

on: push
jobs:
  run:
    runs-on: ubuntu-latest
    steps:
      - uses: iterative/setup-dvc@v1
      - run: |
          git init
          dvc init
          
          dvc remote add origin gdrive://root
          dvc remote modify origin --local gdrive_use_service_account true
          # dvc remote modify origin --local gdrive_service_account_json_file_path /dev/null

          date > file
          dvc add file

          dvc push --verbose --remote origin
        env:
          GDRIVE_CREDENTIALS_DATA: ${{ secrets.ORIGINAL_SERVICE_ACCOUNT_JSON }}

Note: the ${{ secrets.ORIGINAL_SERVICE_ACCOUNT_JSON }} variable represents a GCP Service Account file in the standard JSON format provided by Google, not a DVC credentials JSON file.

Actual output

$ dvc push --verbose --remote origin
2021-06-25 22:04:09,030 ERROR: failed to push data to the cloud - To use service account, set `gdrive_service_account_json_file_path`, and optionally`gdrive_service_account_user_email` in DVC config
Learn more at <https://man.dvc.org/remote/modify>
------------------------------------------------------------
Traceback (most recent call last):
  File "dvc/command/data_sync.py", line 57, in run
  File "dvc/repo/__init__.py", line 51, in wrapper
  File "dvc/repo/push.py", line 44, in push
  File "dvc/data_cloud.py", line 77, in push
  File "dvc/data_cloud.py", line 38, in get_remote
  File "dvc/data_cloud.py", line 58, in _init_remote
  File "dvc/remote/__init__.py", line 8, in get_remote
  File "dvc/fs/gdrive.py", line 128, in __init__
  File "dvc/fs/gdrive.py", line 155, in _validate_config
dvc.exceptions.DvcException: To use service account, set `gdrive_service_account_json_file_path`, and optionally`gdrive_service_account_user_email` in DVC config
Learn more at <https://man.dvc.org/remote/modify>
2021-06-25 22:04:09,033 DEBUG: Analytics is enabled.
------------------------------------------------------------
2021-06-25 22:04:09,076 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmp8tbexj0k']'
2021-06-25 22:04:09,077 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmp8tbexj0k']'

Expected output

$ dvc push --verbose --remote origin
2021-06-25 22:17:58,441 DEBUG: Preparing to upload data to 'gdrive://root'
2021-06-25 22:17:58,441 DEBUG: Preparing to collect status from gdrive://root
2021-06-25 22:17:58,441 DEBUG: Collecting information from local cache...
2021-06-25 22:17:58,441 DEBUG: Collecting information from remote cache...
2021-06-25 22:17:58,441 DEBUG: Matched '0' indexed hashes
2021-06-25 22:17:58,442 DEBUG: Querying 1 hashes via object_exists
2021-06-25 22:17:58,672 DEBUG: GDrive remote auth with config '***'client_config_backend': 'settings', 'client_config_file': 'client_secrets.json', 'save_credentials': True, 'oauth_scope': ['https://www.googleapis.com/auth/drive', 'https://www.googleapis.com/auth/drive.appdata'], 'save_credentials_backend': 'file', 'save_credentials_file': '/home/runner/work/test/test/.dvc/tmp/.5VsCA2nJs9ZQgeBZ8pWNAk.tmp', 'get_refresh_token': True, 'service_config': ***'client_user_email': None, 'client_json_file_path': '/home/runner/work/test/test/.dvc/tmp/.mpgSFHPn7NBgMe35nngenb.tmp'***'.
2021-06-25 22:17:59,308 DEBUG: Uploading '.dvc/cache/5d/afc8549edbdc83c20f11cbfde93cc4' to 'gdrive://root/5d/afc8549edbdc83c20f11cbfde93cc4'
1 file pushed
2021-06-25 22:18:01,059 DEBUG: Analytics is enabled.
2021-06-25 22:18:01,113 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmp0ocbbm6a']'
2021-06-25 22:18:01,116 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmp0ocbbm6a']'

Environment information

$ dvc doctor
DVC version: 2.4.1 (deb)
---------------------------------
Platform: Python 3.8.10 on Linux-5.8.0-1033-azure-x86_64-with-glibc2.7
Supports: All remotes
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: gdrive
Workspace directory: ext4 on /dev/root
Repo: dvc, git