BigQuery: bytes processed should be logged for all dbt commands that perform queries

Describe the feature

With the release of dbt 0.18.0, launching a dbt run command against a bigquery database will output the bytes processed when a model is successfully deployed. However, to get a full picture of the amount of bytes dbt processes on bigquery (and hence to get an idea of the cost of running a specific dbt command), this should also be logged when launching dbt test, dbt source snapshot-freshness and dbt run-operation commands.

Additional context

For dbt run commands, this feature was discussed in #2526, it would be nice to extend this to also log this number for the other commands mentioned above.

Who will this benefit?

This is a bigquery specific issue.

Are you interested in contributing this feature?

Sure, happy to see how we could add this.

[jtcohen6]

#2961 adds an adapter-specific dict to run_results.json. Let’s consider that a partway resolution to this issue; meanwhile, I’ll open a new issue to address the desire for recording bytes_processed for queries from other invocation types—namely dbt test—which we can revisit for v0.20.0.

[jtcohen6]

IIRC, after discussing this live, we decided we’d opt for:

an unstructured dict of result data … populated on a per-adapter basis. The adapter is responsible for generating the status text and any extra result data it might think is appropriate, then that gets decorated with any task-specific result data that might be appropriate.

The trickiness will be in surfacing that data to stdout. While this isn’t a blocker for #2493, it’s highly related. I’m removing “good first issue” and “bigquery”—this gets into our core plumbing—and pulling it into v0.19.

Edit: Eventually, we may want this to be a list of dictionaries (query_stats). One model/materialization may include multiple queries and we’d want to collect stats for all of them. This isn’t something we do a good job of today in run results more generally.