Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

How can I set bigquery streaming insert timeout?

See original GitHub issue

I caused Error. How can I set timeout (read timeout?)?

com.google.cloud.bigquery.BigQueryException: Read timed out
        at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:86)
        at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.insertAll(HttpBigQueryRpc.java:283)
        at com.google.cloud.bigquery.BigQueryImpl.insertAll(BigQueryImpl.java:443)
        at jp.co.xxx.BigQuerySender.insertStreamingData(BigQuerySender.java:68)
        at jp.co.xxx.BigQuerySender.insert(BigQuerySender.java:60)
        at jp.co.xxx.GcpSender.run(GcpSender.java:67)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
        at sun.security.ssl.InputRecord.read(InputRecord.java:503)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
        at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
        at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
        at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)
        at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
        at com.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:37)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:94)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
        at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.insertAll(HttpBigQueryRpc.java:281)

Issue Analytics

State:
Created 6 years ago
Comments:7

Top GitHub Comments

1reaction

vam-googlecommented, Aug 7, 2017

The HttpTransportOptions class allows to set read and connect timeouts. Please check the RemoteBigQueryHelper code for an example of how it can be done:

HttpTransportOptions transportOptions = BigQueryOptions.getDefaultHttpTransportOptions();
transportOptions = transportOptions.toBuilder().setConnectTimeout(60000).setReadTimeout(60000)
    .build();
BigQueryOptions bigqueryOptions = BigQueryOptions.newBuilder()
    .setRetrySettings(retrySettings())
    .setTransportOptions(transportOptions)
    .build();

BigQuery bigquery = bigqueryOptions.getService();
//bigquery.insertAll()

@gyamxxx, please let us know if it resolves your issue.

0reactions

vam-googlecommented, Aug 10, 2017

Yes, feel free to open a new issue (putting a link to this one). Closing this issue as resolved. If you choose to open a new issue, please clarify if you always receive this SSL error, or only sometimes (i.e. is it sporadic or no). If it is sporadic, I suspect the actual proxy you are using may be the problem, not the bigquery service or the client libraries.

Top Results From Across the Web

Quotas and limits | BigQuery - Google Cloud

The time limit for an individual statement within a multi-statement query is 6 hours. Streaming inserts. The following quotas and limits apply when...

How to specify a timeout when streaming rows into BigQuery?

Best recommendation I can give you: Don't stream directly to BigQuery from a process you don't want to block. Set up a service...

Create a Google BigQuery Connection ‒ Qlik Connectors

Query timeout, Amount of time before a data load query times out. Can be set from 30 seconds to 65535 seconds. Default is...

How to Update Table in BigQuery if Streaming Does Not Work

The default insert mode is streaming. When a table is updated via streaming in BigQuery it can take a while (even up to...

BigQuery Writer - Striim

Streaming : Incoming data is buffered locally as one memory buffer per target table. Once the upload condition is met, BigQuery Writer uses...