gsutil not using gcloug oauth details

I can’t seem to get gsutil to use the oAuth2.0 credentials that gcloud is using.

Your "Oauth 2.0 User Account" credentials are invalid. Please run
  $ gcloud auth login
Failure: unauthorized_client.

I used to have gsutil installed via pip and with 4.20 installed, I can do gsutil config and then it works fine. When installing from with the Cloud SDK curl https://sdk.cloud.google.com | bash I then run gcloud auth login go through those steps, and gcloud reports logged in.

Gcloud info

Google Cloud SDK [119.0.0]

Platform: [Mac OS X, x86_64]
Python Version: [2.7.12 (default, Jun 29 2016, 14:05:02)  [GCC 4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.31)]]
Python Location: [/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python]
Site Packages: [Disabled]

Installation Root: [/Users/domster83/google-cloud-sdk]
Installed Components:
  core: [2016.07.21]
  core-nix: [2016.03.28]
  gcloud: []
  gsutil-nix: [4.18]
  gsutil: [4.19]
  bq: [2.0.24]
  bq-nix: [2.0.24]
System PATH: [/Users/domster83/.rbenv/shims:/Users/domster83/google-cloud-sdk/bin:/Users/domster83/laptop/bin:~/bin:/Library/Developer/Toolchains/swift-latest.xctoolchain/usr/bin:usr/local/heroku/bin:/usr/local/sbin:/usr/local/lib/node_modules:/Users/domster83/projects/go/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin]
Cloud SDK on PATH: [True]

Installation Properties: [/Users/domster83/google-cloud-sdk/properties]
User Config Directory: [/Users/domster83/.config/gcloud]
Active Configuration Name: [default]
Active Configuration Path: [/Users/domster83/.config/gcloud/configurations/config_default]

Account: [[e-mail]]
Project: [my-bucket]

Current Properties:
  [core]
    project: [my-bucket]
    account: [[e-mail]]
    disable_usage_reporting: [False]

Logs Directory: [/Users/domster83/.config/gcloud/logs]
Last Log File: [/Users/domster83/.config/gcloud/logs/2016.08.02/15.49.49.740983.log]

gsutil -D

gsutil version: 4.19
checksum: 67da5fbdef140f1663fbb11e96bb9f90 (OK)
boto version: 2.39.0
python version: 2.7.12 (default, Jun 29 2016, 14:05:02) [GCC 4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.31)]
OS: Darwin 16.0.0
multiprocessing available: True
using cloud sdk: True
config path: /Users/domster83/.boto
gsutil path: /Users/domster83/google-cloud-sdk/platform/gsutil/gsutil
compiled crcmod: True
installed via package manager: False
editable install: False
Command being run: /Users/domster83/google-cloud-sdk/platform/gsutil/gsutil -o GSUtil:default_project_id=trilbytv-1191 -D
config_file_list: ['/Users/domster83/.boto']
config: [('debug', '0'), ('working_dir', '/mnt/pyami'), ('https_validate_certificates', 'True'), ('debug', '0'), ('working_dir', '/mnt/pyami'), ('content_language', 'en'), ('default_api_version', '2'), ('default_project_id', 'my-bucket')]

I’ve already updated boto to 2.42.0 via pip, tried reinstalling the cloud-sdk in another location.

I feel like I’m going round in circles.

[thobrla]

The pip install and the gcloud install are separate from each other. If you are using the pip install, then gsutil config will set up your .boto file. But if you are using a gcloud install, then gcloud auth login should be used. It sounds like you are mixing the two configurations? Could you try the following steps:

1. Remove /Users/domster83/.boto
2. Run gcloud auth login
3. Run gsutil

[houglum]

Oh hey, I can totally reproduce this today. I think the shell I was using yesterday had some leftover alias for gsutil such that I was invoking a non-gcloud version. Apologies for the confusion 😃

Anyway, upon realizing that, I figured out that this is a bug with the gsutil bootstrapping script that gcloud uses, /path-to-cloud-sdk/bin/bootstrapping/gsutil.py. Here’s the culprit snippet:

    # We construct a BOTO_PATH that tacks the refresh token config
    # on the end.
    if boto_config:
      boto_path = os.pathsep.join([boto_config, gsutil_path])
    elif boto_path:
      boto_path = os.pathsep.join([boto_path, gsutil_path])
    else:
      path_parts = ['/etc/boto.cfg',
                    os.path.expanduser(os.path.join('~', '.boto')),
                    gsutil_path]
      boto_path = os.pathsep.join(path_parts)

      if 'BOTO_CONFIG' in os.environ:
        del os.environ['BOTO_CONFIG']
      os.environ['BOTO_PATH'] = boto_path

That last line should be shifted left one indentation level so that it gets applied regardless of which predicates above were true. I’ll let our gcloud folks know so we can get this fixed.

(Also, WRT my previous comment, it looks like the gcloud script passes its configured default project as a top-level option, i.e. “-o ‘GSUtil:default_project_id=…’”, so one cannot overwrite the default project using a local .boto file.)