Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Low-level retries on resumable uploads always fail
See original GitHub issueWe’ve been getting errors from the GCS connector when resumable uploads fail: the initial call returns a 503, and the low-level retries obviously get a 410 because the upload id is no longer valid.
These calls should have higher-level retries that generate a new upload ID.
16-08-2018 05:26:11 UTC parameters INFO - 18/08/16 05:26:10 INFO com.google.cloud.hadoop.util.RetryHttpInitializer: Encountered status code 503 when accessing URL https://www.googleapis.com/upload/storage/v1/b/<bucket>/o?ifGenerationMatch=0&name=<file>&uploadType=resumable&upload_id=<upload id>. Delegating to response handler for possible retry.
16-08-2018 05:26:12 UTC parameters INFO - 18/08/16 05:26:11 INFO com.google.cloud.hadoop.util.RetryHttpInitializer: Encountered status code 410 when accessing URL https://www.googleapis.com/upload/storage/v1/b/<bucket>/o?ifGenerationMatch=0&name=<path>&uploadType=resumable&upload_id=<upload id>. Delegating to response handler for possible retry.
16-08-2018 05:26:12 UTC parameters INFO - 18/08/16 05:26:11 ERROR com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel: Exception not convertible into handled response
16-08-2018 05:26:12 UTC parameters INFO - com.google.api.client.googleapis.json.GoogleJsonResponseException: 410 Gone
16-08-2018 05:26:12 UTC parameters INFO - {
16-08-2018 05:26:12 UTC parameters INFO - "code" : 503,
16-08-2018 05:26:12 UTC parameters INFO - "errors" : [ {
16-08-2018 05:26:12 UTC parameters INFO - "domain" : "global",
16-08-2018 05:26:12 UTC parameters INFO - "message" : "Backend Error",
16-08-2018 05:26:12 UTC parameters INFO - "reason" : "backendError"
16-08-2018 05:26:12 UTC parameters INFO - } ],
16-08-2018 05:26:12 UTC parameters INFO - "message" : "Backend Error"
16-08-2018 05:26:12 UTC parameters INFO - }
Issue Analytics
- State:
- Created 5 years ago
- Comments:14 (6 by maintainers)
Top Results From Across the Web
Low-level retries on resumable uploads always fail · Issue #121
We've been getting errors from the GCS connector when resumable uploads fail: the initial call returns a 503, and the low-level retries obviously...
Read more >Resumable uploads | Cloud Storage - Google Cloud
A resumable upload allows you to resume data transfer operations to Cloud Storage after a communication failure has interrupted the flow of data....
Read more >Global Flags - Rclone
Global Flags. This describes the global flags available to every rclone command split into two groups, non backend and backend flags.
Read more >Test Retries - Cypress Documentation
Only upload videos for specs with failing or retried tests. The example below shows how to delete the recorded video for specs that...
Read more >Uploading an object using multipart upload - Amazon Simple ...
If you need to pause and resume multipart uploads, vary part sizes during the upload, or do not know the size of the...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Thanks @medb . I’ve done a crap-ton of tests looking at concurrency count and buffer sizes and have settled on 32mb with our default concurrent. If you get the chunk size too big, the threads have to wait too long and your concurrency averages drop. I’ve also done tests around memory versus disk buffering (local HDD versus SSD) and other settings.
@j256, in this case you may try to increase upload chunk to 128 MiB - it will slightly increase performance and will reduce numbed of files that you upload from 2.67m to 1.335m which could improve reliability: