Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

guide: `dvc exp push/pull` with GDrive service accounts

See original GitHub issue

Bug Report

dvc exp push/pull: google service account rate limits

Description

Google service accounts have upload/download limits (which are quite low - at least by standard). If these limits are exceeded pushing/pulling experiments to the remote throws a 403 http error ‘The user has exceeded their Drive storage quota’. This issue can be fixed with delegation and setting gdrive_service_account_user_email to be the user email (not service account email) as proposed by @shcheklein. This trick is not mentioned anywhere in the documentation, however.

Not having to rely on delegation would be optimal?

Reproduce

  1. dvc exp push origin exp-1b1a1

Output of dvc doctor:

DVC version: 2.22.0 (pip)
---------------------------------
Platform: Python 3.8.13 on Linux-5.4.0-125-generic-x86_64-with-glibc2.17
Supports:
        gdrive (pydrive2 = 1.14.0),
        http (aiohttp = 3.8.1, aiohttp-retry = 2.5.2),
        https (aiohttp = 3.8.1, aiohttp-retry = 2.5.2)
Cache types: hardlink, symlink
Cache directory: nfs on home:/vol/home0
Caches: local
Remotes: gdrive
Workspace directory: nfs on home:/vol/home0
Repo: dvc, git

Other

Running exp push -v result in the following strack trace:

(thesis-venv) (base) zombie case_similarity 1007 % dvc exp push origin exp-1b1a1 -v
2022-09-12 20:54:50,823 DEBUG: git push experiment '['refs/exps/47/7635b282555377e99b41ef040a7337fc166ea3/exp-1b1a1:refs/exps/47/7635b282555377e99b41ef040a7337fc166ea3/exp-1b1a1']' -> 'origin'
2022-09-12 20:54:52,957 DEBUG: dvc push experiment '[ExpRefInfo(baseline_sha='477635b282555377e99b41ef040a7337fc166ea3', name='exp-1b1a1')]'
2022-09-12 20:54:53,727 DEBUG: Preparing to transfer data from '/m/home/home0/07/porkhom1/data/Desktop/case_similarity/.dvc/cache' to '1YxTXT0Y8GIwb186t1pnkgiYZR0_YJ6bJ'
2022-09-12 20:54:53,727 DEBUG: Preparing to collect status from '1YxTXT0Y8GIwb186t1pnkgiYZR0_YJ6bJ'
2022-09-12 20:54:53,728 DEBUG: Collecting status from '1YxTXT0Y8GIwb186t1pnkgiYZR0_YJ6bJ'
2022-09-12 20:54:55,145 DEBUG: Querying 8 oids via object_exists
2022-09-12 20:55:05,713 DEBUG: Querying 0 oids via object_exists
2022-09-12 20:55:06,134 DEBUG: Estimated remote size: 768 files
2022-09-12 20:55:06,135 DEBUG: Querying '6' oids via traverse
2022-09-12 20:55:09,326 DEBUG: Preparing to collect status from '/m/home/home0/07/porkhom1/data/Desktop/case_similarity/.dvc/cache'
2022-09-12 20:55:09,327 DEBUG: Collecting status from '/m/home/home0/07/porkhom1/data/Desktop/case_similarity/.dvc/cache'
2022-09-12 20:55:10,155 ERROR: failed to transfer 'md5: ae359fec36097f0af096e8dac9326040' - <HttpError 403 when requesting None returned "The user has exceeded their Drive storage quota". Details: "[{'domain': 'usageLimits', 'reason': 'quotaExceeded', 'message': 'The user has exceeded their Drive storage quota'}]">
------------------------------------------------------------
Traceback (most recent call last):
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/pydrive2/files.py", line 678, in _FilesInsert
    self.auth.service.files()
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/googleapiclient/http.py", line 902, in execute
    _, body = self.next_chunk(http=http, num_retries=num_retries)
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/googleapiclient/http.py", line 1022, in next_chunk
    raise ResumableUploadError(resp, content)
googleapiclient.errors.ResumableUploadError: <HttpError 403 when requesting None returned "The user has exceeded their Drive storage quota". Details: "[{'domain': 'usageLimits', 'reason': 'quotaExceeded', 'message': 'The user has exceeded their Drive storage quota'}]">

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/dvc_data/transfer.py", line 31, in wrapper
    func(path, *args, **kwargs)
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/dvc_data/transfer.py", line 170, in func
    return dest.add(
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/dvc_data/hashfile/db.py", line 77, in add
    super().add(path, fs, oid, hardlink=hardlink, callback=callback)
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/dvc_objects/db.py", line 122, in add
    generic.transfer(
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/dvc_objects/fs/generic.py", line 100, in transfer
    _try_links(
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/dvc_objects/fs/generic.py", line 63, in _try_links
    return copy(from_fs, from_path, to_fs, to_path, callback=callback)
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/dvc_objects/fs/generic.py", line 39, in copy
    return to_fs.put_file(from_path, to_path, callback=callback)
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/dvc_objects/fs/base.py", line 369, in put_file
    self.fs.put_file(
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/pydrive2/fs/spec.py", line 525, in put_file
    self.upload_fobj(stream, rpath, callback=callback)
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/pydrive2/fs/spec.py", line 517, in upload_fobj
    return self._gdrive_upload_fobj(
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/funcy/decorators.py", line 45, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/funcy/flow.py", line 127, in retry
    return call()
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/funcy/decorators.py", line 66, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/pydrive2/fs/spec.py", line 533, in _gdrive_upload_fobj
    item.Upload()
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/pydrive2/files.py", line 512, in Upload
    self._FilesInsert(param=param)
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/pydrive2/auth.py", line 84, in _decorated
    return decoratee(self, *args, **kwargs)
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/pydrive2/files.py", line 683, in _FilesInsert
    raise ApiRequestError(error)
pydrive2.files.ApiRequestError: <HttpError 403 when requesting None returned "The user has exceeded their Drive storage quota". Details: "[{'domain': 'usageLimits', 'reason': 'quotaExceeded', 'message': 'The user has exceeded their Drive storage quota'}]">
------------------------------------------------------------
2022-09-12 20:55:10,157 ERROR: failed to transfer 'md5: 0293b8ba56ec051249d8dc48460d2323' - <HttpError 403 when requesting None returned "The user has exceeded their Drive storage quota". Details: "[{'domain': 'usageLimits', 'reason': 'quotaExceeded', 'message': 'The user has exceeded their Drive storage quota'}]">
------------------------------------------------------------
Traceback (most recent call last):
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/pydrive2/files.py", line 678, in _FilesInsert
    self.auth.service.files()
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/googleapiclient/http.py", line 902, in execute
    _, body = self.next_chunk(http=http, num_retries=num_retries)
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/googleapiclient/http.py", line 1022, in next_chunk
    raise ResumableUploadError(resp, content)
googleapiclient.errors.ResumableUploadError: <HttpError 403 when requesting None returned "The user has exceeded their Drive storage quota". Details: "[{'domain': 'usageLimits', 'reason': 'quotaExceeded', 'message': 'The user has exceeded their Drive storage quota'}]">

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/dvc_data/transfer.py", line 31, in wrapper
    func(path, *args, **kwargs)
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/dvc_data/transfer.py", line 170, in func
    return dest.add(
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/dvc_data/hashfile/db.py", line 77, in add
    super().add(path, fs, oid, hardlink=hardlink, callback=callback)
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/dvc_objects/db.py", line 122, in add
    generic.transfer(
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/dvc_objects/fs/generic.py", line 100, in transfer
    _try_links(
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/dvc_objects/fs/generic.py", line 63, in _try_links
    return copy(from_fs, from_path, to_fs, to_path, callback=callback)
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/dvc_objects/fs/generic.py", line 39, in copy
    return to_fs.put_file(from_path, to_path, callback=callback)
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/dvc_objects/fs/base.py", line 369, in put_file
    self.fs.put_file(
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/pydrive2/fs/spec.py", line 525, in put_file
    self.upload_fobj(stream, rpath, callback=callback)
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/pydrive2/fs/spec.py", line 517, in upload_fobj
    return self._gdrive_upload_fobj(
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/funcy/decorators.py", line 45, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/funcy/flow.py", line 127, in retry
    return call()
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/funcy/decorators.py", line 66, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/pydrive2/fs/spec.py", line 533, in _gdrive_upload_fobj
    item.Upload()
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/pydrive2/files.py", line 512, in Upload
    self._FilesInsert(param=param)
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/pydrive2/auth.py", line 84, in _decorated
    return decoratee(self, *args, **kwargs)
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/pydrive2/files.py", line 683, in _FilesInsert
    raise ApiRequestError(error)
pydrive2.files.ApiRequestError: <HttpError 403 when requesting None returned "The user has exceeded their Drive storage quota". Details: "[{'domain': 'usageLimits', 'reason': 'quotaExceeded', 'message': 'The user has exceeded their Drive storage quota'}]">
------------------------------------------------------------
2022-09-12 20:55:10,174 ERROR: 2 files failed to upload
------------------------------------------------------------
Traceback (most recent call last):
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/dvc/data_cloud.py", line 88, in transfer
    return transfer(src_odb, dest_odb, objs, **kwargs)
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/dvc_data/transfer.py", line 184, in transfer
    _do_transfer(
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/dvc_data/transfer.py", line 119, in _do_transfer
    raise TransferError(total_fails)
dvc_data.transfer.TransferError: 2 transfer failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/dvc/repo/push.py", line 68, in push
    pushed += self.cloud.push(
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/dvc/data_cloud.py", line 109, in push
    return self.transfer(
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/dvc/data_cloud.py", line 90, in transfer
    raise FileTransferError(exc.fails) from exc
dvc.exceptions.FileTransferError: 2 files failed to transfer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/dvc/cli/__init__.py", line 185, in main
    ret = cmd.do_run()
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/dvc/cli/command.py", line 22, in do_run
    return self.run()
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/dvc/commands/experiments/push.py", line 27, in run
    pushed_exps = self.repo.experiments.push(
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/dvc/repo/experiments/__init__.py", line 529, in push
    return push(self.repo, *args, **kwargs)
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/dvc/repo/scm_context.py", line 156, in run
    return method(repo, *args, **kw)
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/dvc/repo/experiments/push.py", line 73, in push
    _push_cache(repo, push_cache_ref, **kwargs)
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/dvc/repo/experiments/push.py", line 122, in _push_cache
    repo.push(jobs=jobs, remote=dvc_remote, run_cache=run_cache, revs=revs)
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "/m/home/home0/07/porkhom1/data/Desktop/case_similarity/thesis-venv/lib/python3.8/site-packages/dvc/repo/push.py", line 72, in push
    raise UploadError(exc.amount)
dvc.exceptions.UploadError: 2 files failed to upload
------------------------------------------------------------
2022-09-12 20:55:10,195 DEBUG: Analytics is enabled.
2022-09-12 20:55:10,300 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpu8a__wm9']'
2022-09-12 20:55:10,302 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpu8a__wm9']'
(thesis-venv) (base) zombie case_similarity 1008 %

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:9 (8 by maintainers)

github_iconTop GitHub Comments

1reaction
JohnTheDeerecommented, Oct 27, 2022

Entropy is always growing, but resistance is not futile!

Some kind of a FAQ like section would be good.

This is more of a general comment regarding the google page but one option could be to have a table at the top with the different options to integrate dvc with google (pro’s/con’s etc) such that each ”option” links to its own section. That would ”decouple” different options a bit and make it easier for the user to pick one option and follow that.

When I first started using dvc with google what I eventually ended up doing was (I remembered) there was a youtube video on your youtube page explaining how to set things up. One problem was it was a bit difficult to remember which video and where the particular info I needed was mentioned. Not optimal.

Youtube tutorials are good - usually the problem is that they are too long (10 min video and the info I am looking for is 30 s). One idea could be to create VERY short youtube instructional videos. Litterarily 30s long which would be fast and show how to set things up. As the battle for attention just keeps getting more intense (youtube shorts/tiktok) I think something like this would be very user friendly.

1reaction
shchekleincommented, Oct 27, 2022

The whole point that it was not obvious at all that e.g. service account doesn’t have the same amount of storage available as a regular user account. I think this is important to clarify and provide a workaround. Potentially, I would even consider creating a troubleshooting section on that page with some “situations”. I see that people keep hitting issues with credentials, etc, etc.

Read more comments on GitHub >

github_iconTop Results From Across the Web

guide: dvc exp push/pull with GDrive service accounts #3963
This issue can be fixed with delegation and setting gdrive_service_account_user_email to be the user email (not service account email) as ...
Read more >
How to Setup a Google Drive DVC Remote
In this guide we explain the existing ways to setup Google Drive remote ... Having your own GC project, it's also possible to...
Read more >
Untitled
K to 12 curriculum guide filipino 2, Jak dojsc na moczary rybiego oka, ... Unaitas sacco branches, The kuiper belt refers to, Gface...
Read more >
Browser built in viewer - device.report
We pledge ours e lves to con tinue to improve our service, to increase our stoc ks ... Full size 6-tube amplifier with...
Read more >
Untitled
Mygica atv586 bluetooth, I am never drinking again, Bathroom shower ideas, Dell 2130 service manual, Nenu sailaja hd ringtones, Flame 100 safety burner, ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

guide: clarify that some Hydra features are always on

Next page

Improve Twitter integration