Logging: parse entries created by bigquery_resource
See original GitHub issueclient_logging.list_entries()
on BigQuery log entries renders a list of ProtobufEntry
objects.
Unfortunately there is no documentation on how to read (“unpack” ?) the payload_pb
property, for instance to retrieve the jobId
.
Request to extent the documentation beyond retrieving entries that were written to a logger, and include snippets on how-to parse protoPayload
messages like those created by GCP services.
Issue Analytics
- State:
- Created 5 years ago
- Comments:6 (3 by maintainers)
Top Results From Across the Web
View logs routed to BigQuery - Google Cloud
This document explains how you can find log entries that you routed from Cloud Logging to BigQuery tables. Logging sinks stream logging data...
Read more >Audit logging for Firebase Management - Google
This page describes the audit logs created by Firebase as part of Cloud Audit Logs. ... like resource.type ; for details, review Find...
Read more >json - Export Cloud Logging to Big Query Auto-Parse into ...
In your case, if you edit the configuration file that is capturing your logs (under /etc/google-fluentd/config.d/), configure 'format json', ...
Read more >The Complete Guide to Log Analysis with Big Query - Brainlabs
How important does Google see parts of your site? We're started to get more complex now. Looking into your log files can also...
Read more >How to export logs from Google Cloud Logging to BigQuery
Google Cloud Blog → http://goo.gle/2NEnP2O Getting Started with Google Cloud Logging → https://goo.gle/3ox6N3k What is BigQuery ...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Not sure why it gets closed as what seems to be missing is the module compiled from proto, as per https://stackoverflow.com/questions/50301632/using-python-to-query-gcp-stackdriver-logs
Instead of each of us compiling a python module for it, it would make sense to have a new google.cloud.audit package containing, at a minimum, the AuditLog module. After that, I agree @tseaver’s code makes it straightforward to parse.
as a fully fleshed out example, here’s how i generated my own
request_log_pb2.py
file based onrequest_log.proto
to parsegoogle.appengine.logging.v1.RequestLog
serialized protobufs: https://stackoverflow.com/a/59414384/186123. more background in #2572.@yiga2 is right, though, Google should ideally maintain official versions of these Python protobuf bindings instead.