allow access token to be passed in to `gsutil cp`

The current access token can be obtained by doing a gcloud auth print-access-token. That works for service accounts too. It would be useful if the access token can be passed in for commands, especially gsutil cp.

As a workaround, I wrote this script to download files:

#!/usr/bin/env python
# derived from https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/storage/api/crud_object.py

import argparse
import json
import io

from googleapiclient import discovery
from googleapiclient import http

from oauth2client.client import GoogleCredentials
from oauth2client.client import AccessTokenCredentials

def main(bucket, filename, accesstoken):
    service = create_service(accesstoken)
    with io.FileIO(filename, mode='wb') as outfile:
        get_object(service, bucket, filename, outfile)

def create_service(accesstoken):
    # print("accesstoken: %s" % accesstoken)
    if not accesstoken:
        credentials = GoogleCredentials.get_application_default()
    else:
        credentials = AccessTokenCredentials(accesstoken, 'my-user-agent/1.0')
    return discovery.build('storage', 'v1', credentials=credentials)

def get_object(service, bucket, filename, outfile):
    print("gsutil cp gs://%s/%s %s" % (bucket, filename, filename))
    req = service.objects().get_media(bucket=bucket, object=filename)
    downloader = http.MediaIoBaseDownload(outfile, req)
    done = False
    while done is False:
        status, done = downloader.next_chunk()
        # print("Download {}%.".format(int(status.progress() * 100)))
    print("gsutil cp done")

if __name__ == '__main__':
    parser = argparse.ArgumentParser(
        description=__doc__,
        formatter_class=argparse.RawDescriptionHelpFormatter)
    parser.add_argument('bucket', help='storage bucket')
    parser.add_argument('filename', help='the file to download')
    parser.add_argument('accesstoken', help='access token')
    args = parser.parse_args()
    main(args.bucket, args.filename, args.accesstoken)

[melissachang]

I also am trying to use gsutil without Service Account key. (My employer prohibits use of SA keys. Ironically, my employer is Google.) Documenting what worked.

There are 3 versions of gsutil:

1. Standalone gsutil. This is the oldest, and predates gcloud.
2. gsutil as part of gcloud
3. gcloud alpha storage. This is the newest. Faster than 2.

For 2: Write the ~/.config/gcloud/legacy_credentials/default/.boto that google-cloud-sdk/bin/bootstrapping/gsutil.py expects:

[OAuth2]
client_id = <CLIENT_ID>
client_secret = <CLIENT_SECRET>

[Credentials]
gs_oauth2_refresh_token = <REFRESH_TOKEN>

For 3: Do what rest of gcloud does: set CLOUDSDK_AUTH_ACCESS_TOKEN. Note that you’ll need Python 3 installed. If you don’t have Python 3, you’ll get TypeError: 'instance has no next() method'.

[ctaggart]

I used mitmproxy to troubleshoot the request and found out I was just missing the alt=media query param. I’m closing this and will just use curl for docker builds.

curl -o $StorageFile -H "Authorization: Bearer $StorageAccessToken" https://www.googleapis.com/storage/v1/b/$StorageBucket/o/$StorageFile?alt=media