intermittent error 400 when calling bigquery API

I’m experiencing a cryptic error message while running a long bigquery job:

  File "/usr/local/lib/python3.6/site-packages/googleapiclient/http.py", line 851, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://www.googleapis.com/bigquery/v2/projects/<project-name>/queries/job_<id>?pageToken=<page_token>&alt=json returned "{0}">

The point of attention is the returned "{0}" part of the message where the reason was supposed to be. When looking at the job history with the bq --format=prettyjson show -j <jobid> command it also doesn’t show the reason field, so that may be the root cause.

Environment details

• Python version: 3.6.5
• google-api-python-client: 1.7.8
• google-cloud-bigquery: 1.11.2
• google-cloud-storage: 1.15.0

Steps to reproduce

This error occurs somewhat randomly while processing a long running bigquery job.

After retrying a few times the job usually succeeds. I wonder if this is related with cause resourcesExceeded documented in https://cloud.google.com/bigquery/troubleshooting-errors. The other possible causes doesn’t seem to be related.

[shollyman]

Checked this morning, it looks like the backend changes to address the malformed response have been deployed to tasks in all regions. Cases where requests were getting the 400 response and the {0} body were representative of a backend component where error propagation was not being handled correctly. Now such conditions should yield an appropriate retryable HTTP error, and a properly formatted response.

@danicat Thanks again for reporting this, the additional details you were able to provide about specific requests were helpful in tracking this down.

[shollyman]

@danicat It looks like the BigQuery engineering team has identified a situation in the tabledata backend that can trigger the unusual 400 response with the poorly formatted response. No details/ETA on resolution yet, but thanks for the report helping to identify this. I’ll keep this issue open until the internal issue is resolved.