Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

gsutil rsync hangs on large files

See original GitHub issue

Hi! We are using gsutil rsync to upload our backups to coldline storage. After our files become larger (more than 30GB) rsync starts hanging on each such file. The same situation is for gsutil cp. I’ve already sent a log file with gsutil -D rsync to gs-team@google.com

gsutil version: 4.25
boto version: 2.42.0
python version: 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609]
OS: Linux 4.4.0-77-generic
multiprocessing available: True
using cloud sdk: True
config path(s): /etc/boto.cfg
gsutil path: /usr/lib/google-cloud-sdk/platform/gsutil/gsutil
compiled crcmod: True
installed via package manager: False
editable install: False

Issue Analytics

State:
Created 6 years ago
Reactions:5
Comments:14

Top GitHub Comments

3reactions

fabio-rigatocommented, Feb 20, 2020

Hello, I’m trying to rsync from gcs to s3 buckets but some files are bigger than 5 GiB.

I receive this error: “exceeds the maximum gsutil-supported size for an S3 upload. S3 objects greater than 5 GiB in size require multipart uploads, which gsutil does not support.”

As you can see here https://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html there is a limitation in S3 where you have to use multipart upload for files bigger than 5 GiB.

Please help me, how can I fix that?

Any help will be really appreciated! thanks @houglum for your time

Best Regards Fabio Rigato

0reactions

fayas-akramcommented, Aug 5, 2022

I am also facing same issue