question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

gcsfs can't access google storage file, switches to anon user

See original GitHub issue

I am trying to read a csv file present in gs bucket directly into dataframe. I used gcloud auth login to authorize account. But when I try to use gcsfs it thorws: gcsfs.utils.HttpError: Anonymous caller does not have storage.objects.list access to <my-bucket>

Now I ca list/download my files using gsutil. I can also list these files by passing adc file as token to fs=gcsfs.GCSFileSystem(token='adc.json'). My issue is how do I set it as default. Is there a particular location where I put this file. I couldn’t find anything in docs.

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:14 (6 by maintainers)

github_iconTop GitHub Comments

6reactions
Gabshacommented, Apr 1, 2020

See here for how google.auth tries to read your credentials. Enabling application-default worked for me: gcloud auth application-default login

5reactions
vlasenkoalexeycommented, May 6, 2020

#255 should fix this issue.

Following code now works just fine:

from gcsfs.core import GCSFileSystem
gcs = GCSFileSystem('<project_id>', token='/Users/<username>/.config/gcloud/application_default_credentials.json')
gcs.ls('<bucket>') 

FYI, to use default GCP authentication fallback mechanism you can use following:

from gcsfs.core import GCSFileSystem
import google.auth

credentials, _ =google.auth.default()
gcs = GCSFileSystem('<project_id>', token=credentials)
gcs.ls('<bucket>') 
Read more comments on GitHub >

github_iconTop Results From Across the Web

gcsfs can't access google storage file, switches to anon user
I am trying to read a csv file present in gs bucket directly into dataframe. I used gcloud auth login to authorize account....
Read more >
google.cloud.storage allows access but gcsfs does not
To open a file using '“google.cloud.storage'” instead of "gcsfs" you can use the below method, #create storage client storage_client ...
Read more >
GCSFs Documentation - Read the Docs
A pythonic file-system interface to Google Cloud Storage. This software is beta, use at your own risk. Please file issues and requests on ......
Read more >
Troubleshooting | Cloud Storage
This page describes troubleshooting methods for common errors you may encounter while using Cloud Storage. See the Google Cloud Status Dashboard for ...
Read more >
GCSFS — GCSFs 2022.11.0+2.g111769a documentation
A pythonic file-system interface to Google Cloud Storage. This software is beta, use at your own risk. Please file issues and requests on...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found