Problems uploading files to Cloud Storage: com.google.cloud.storage.StorageException: Error writing request body to server

We randomly get the exception stated below when uploading files to Cloud Storage. The file size does not seem to be the relevant factor - the files where we see the problems are in the range between 25MB and 22GB.

com.google.cloud.storage.StorageException: Error writing request body to server
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:220)
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.write(HttpStorageRpc.java:704)
	at com.google.cloud.storage.BlobWriteChannel$1.run(BlobWriteChannel.java:51)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89)
	at com.google.cloud.RetryHelper.run(RetryHelper.java:74)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
	at com.google.cloud.storage.BlobWriteChannel.flushBuffer(BlobWriteChannel.java:47)
	at com.google.cloud.BaseWriteChannel.flush(BaseWriteChannel.java:122)
	at com.google.cloud.BaseWriteChannel.write(BaseWriteChannel.java:149)
	at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
	at java.nio.channels.Channels.writeFully(Channels.java:101)
	at java.nio.channels.Channels.access$000(Channels.java:61)
	at java.nio.channels.Channels$1.write(Channels.java:174)
	at smarter.ecommerce.commons.io.DelegatingOutputStream.write(DelegatingOutputStream.java:27)
	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1793)
	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769)
	at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744)
        .......
        Caused by: java.io.IOException: Error writing request body to server
	at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3518)
	at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3501)
	at com.google.api.client.util.ByteStreams.copy(ByteStreams.java:55)
	at com.google.api.client.util.IOUtils.copy(IOUtils.java:94)
	at com.google.api.client.http.AbstractInputStreamContent.writeTo(AbstractInputStreamContent.java:72)
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:80)
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.write(HttpStorageRpc.java:685)
	... 85 more

We already had a look at the integrated retry handling and it looks like that exceptions of this kind are not retried when sending a chunk to Cloud Storage. Unfortunately it’s very hard to reproduce with a test. But the problem occurs randomly, but frequently (multiple times a day)!

This might be related: https://stackoverflow.com/questions/50019241/google-cloud-storage-throwing-error-for-multiple-large-file-uploads-concurrently

The code that we are using is pretty much the same as stated in the linked stackoverflow article.

[nitishkumar71]

I don’t think this is an issue of access within google cloud or outside of it. because in case of stackoverflow question, access was being done from Google Cloud itself.

[zguth]

Yes, it’s here: https://github.com/GoogleCloudPlatform/google-cloud-java/issues/3678