How to check current BigQuery usage

Whilst making https://github.com/ofek/pypinfo/pull/29 I got this:

    raise exceptions.from_http_response(response)
google.api_core.exceptions.Forbidden: 403 GET https://www.googleapis.com/bigquery/v2/projects/pypinfo-hugovk/queries/<snip>?maxResults=0&timeoutMs=10000: Quota exceeded: Your project exceeded quota for free query bytes scanned. For more information, see https://cloud.google.com/bigquery/troubleshooting-errors

I can’t find how much of my free monthly TB I’ve used up on the Google console.

After some digging in the console I found this:

https://console.cloud.google.com/home/activity?project=pypinfo-hugovk&authuser=1

Linking to this:

https://console.cloud.google.com/iam-admin/quotas?project=pypinfo-hugovk&authuser=1

Which isn’t very informative – all zeroes and dashes!

I get the error after doing several pypinfo --percent --pip pypinfo pyversion, but pypinfo --percent --pip -d 1 pypinfo is still fine.

Any idea what “QUERY-MBYTES-FOR-UNBILLED-PROJECTS-per-project” really means?

And where to check the monthly 1TB quota?

Thanks!

[ofek]

I added these stats just now in 9.0.0 🙂

Though, I still can’t find a way to see one’s remaining free quota.

[hugovk]

Visit a page something like this, replace TODO_YOUR_PROJECT_HERE with your Pypinfo project name you chose in step 3 of the README:

https://console.cloud.google.com/home/activity?c=Configuration,Data_Access,Development,Monitoring,Platform,Recommendation&project=TODO_YOUR_PROJECT_HERE&authuser=1

On the filter on the right, make sure your project is selected in the dropdown, and enable all categories:

You should then see a bunch of logs like this:

The “Completed job” entries are the most interesting, showing the query pypinfo sent (you can see it locally with pypinfo --test), total billed bytes and total processed bytes:

From here, you can see that re-running a query that has been previously cached doesn’t cost any extra bytes.