question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Logging: parse entries created by bigquery_resource

See original GitHub issue

client_logging.list_entries() on BigQuery log entries renders a list of ProtobufEntry objects.

Unfortunately there is no documentation on how to read (“unpack” ?) the payload_pb property, for instance to retrieve the jobId.

Request to extent the documentation beyond retrieving entries that were written to a logger, and include snippets on how-to parse protoPayload messages like those created by GCP services.

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Comments:6 (3 by maintainers)

github_iconTop GitHub Comments

2reactions
yan-hiccommented, Jun 21, 2018

Not sure why it gets closed as what seems to be missing is the module compiled from proto, as per https://stackoverflow.com/questions/50301632/using-python-to-query-gcp-stackdriver-logs

Instead of each of us compiling a python module for it, it would make sense to have a new google.cloud.audit package containing, at a minimum, the AuditLog module. After that, I agree @tseaver’s code makes it straightforward to parse.

0reactions
snarfedcommented, Dec 19, 2019

as a fully fleshed out example, here’s how i generated my own request_log_pb2.py file based on request_log.prototo parse google.appengine.logging.v1.RequestLog serialized protobufs: https://stackoverflow.com/a/59414384/186123. more background in #2572.

@yiga2 is right, though, Google should ideally maintain official versions of these Python protobuf bindings instead.

Read more comments on GitHub >

github_iconTop Results From Across the Web

View logs routed to BigQuery - Google Cloud
This document explains how you can find log entries that you routed from Cloud Logging to BigQuery tables. Logging sinks stream logging data...
Read more >
Audit logging for Firebase Management - Google
This page describes the audit logs created by Firebase as part of Cloud Audit Logs. ... like resource.type ; for details, review Find...
Read more >
json - Export Cloud Logging to Big Query Auto-Parse into ...
In your case, if you edit the configuration file that is capturing your logs (under /etc/google-fluentd/config.d/), configure 'format json', ...
Read more >
The Complete Guide to Log Analysis with Big Query - Brainlabs
How important does Google see parts of your site? We're started to get more complex now. Looking into your log files can also...
Read more >
How to export logs from Google Cloud Logging to BigQuery
Google Cloud Blog → http://goo.gle/2NEnP2O Getting Started with Google Cloud Logging → https://goo.gle/3ox6N3k What is BigQuery ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found