gsutil cp reports OK but file does not exist in bucket
See original GitHub issueHello,
I’m trying to recursively copy data from a local storage to a Google Storage bucket using gsutil cp
The command issued is
gsutil -m cp -R -L gsutil.log /root/backup/ gs://MY_BUCKET_NAME/
Even the log file shows OK for all files - here is a line of a missing file from the log
file:///root/backup/2015-10-06T181356Z/2015-10-06T181356Z-full/bucket-simplehms/node-167.114.225.106%3A8091/data-0000.cbb,gs://MY_BUCKET_NAME/backup/2015-10-06T181356Z/2015-10-06T181356Z-full/bucket-simplehms/node-167.114.225.106%3A8091/data-0000.cbb,2015-10-06T18:21:07.929552Z,2015-10-06T18:21:09.923508Z,+cKX5JW5ddtJgzK34S9A7g==,,38612992,38612992,OK,
even though everything reports the file transfer as OK some of the files (i.e. the one in the log above) do not exist in the Google Storage bucket.
I’ve tried gsutil ls and the Google Developers Console to ensure it’s not a display issue, it’s not visible in either of them - it just shows an empty folder …
Issue Analytics
- State:
- Created 8 years ago
- Comments:8 (3 by maintainers)
Top Results From Across the Web
cp - Copy files and objects | Cloud Storage - Google Cloud
Description. The gsutil cp command allows you to copy data between your local file system and the cloud, within the cloud, and between...
Read more >Gsutil cp is not copying files - Stack Overflow
I am trying to use the command line on my local machine (anaconda prompt) to download a folder from a google cloud bucket....
Read more >Download and export monthly reports - Play Console Help
Download reports from Google Cloud Storage. Reports are available from Google Cloud Storage. Reports are generated daily and accumulated in monthly CSV files....
Read more >gslib/commands/cp.py - external/gsutil/src - Git at Google
if gs://my_bucket/subdir does not exist, this same gsutil cp command will. create objects named like gs://my_bucket/subdir/a/b/c.
Read more >copy_helper.py - EECS: www-inst.eecs.berkeley.edu
For example, when running the command: gsutil cp file gs://bucket/abc if there's no existing gs://bucket/abc bucket subdirectory we should copy file to the ......
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
similar happens for me, steps: -I create a file with my own user (gsutil cp) | OK -I delete the file from the bucket with my own user (gsutil rm) | OK -a service user tries to create another file with the very same name | gsutil reports all good, but the file never appears in the bucket
It turns out this was an issue in the Google Cloud Platform UI where %3A on the URL was improperly being translated into a colon in the URL. Until that’s fixed, the workaround in the UI is to manually percent-encode the percent sign in the URL, i.e., replace %3A with %253A