Support Google Storage Buckets as output directory

• Weights and Biases version: 0.8.13
• Python version: 3.7
• Operating System: Debian

Description

On Google Cloud, gs:// buckets are not recognised by wandb.init(..., dir="gs://etc/etc")

Reading and writing to buckets requires doing something like this: https://cloud.google.com/appengine/docs/standard/python/googlecloudstorageclient/read-write-to-cloud-storage#writing_to_cloud_storage

Or using tf.gfile

Since TPUs can only read and write checkpoints to gs this prevents us from using wanb whenever we want to use TPU hardware.

[aidangomez]

So when using TF w/ TPUs you actually can’t read or write data + checkpoints locally, it’s all done with gs:// (there isn’t an alternative).

So we could transfer checkpoints and summaries from the bucket to the local machine and then sync with wandb, but if I have a model with 5GB checkpoint files I don’t want to have to pause training to perform that large of a sync, so we’ll need to write logic to do this asynchronously.

I think that since read/write from gs buckets is considered standard for TF, wandb stands to gain a great deal from supporting it. The overhead of writing and incorporating into a training script a daemon syncing between gs:// and the local fs means that a lot of people simply won’t use wandb.

Hopefully it will be prioritised at some point to improve usability of wandb for TPU users.

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the label enhancement to this issue, with a confidence of 0.91. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.