question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

gsutil cp reports OK but file does not exist in bucket

See original GitHub issue

Hello,

I’m trying to recursively copy data from a local storage to a Google Storage bucket using gsutil cp

The command issued is gsutil -m cp -R -L gsutil.log /root/backup/ gs://MY_BUCKET_NAME/

Even the log file shows OK for all files - here is a line of a missing file from the log

file:///root/backup/2015-10-06T181356Z/2015-10-06T181356Z-full/bucket-simplehms/node-167.114.225.106%3A8091/data-0000.cbb,gs://MY_BUCKET_NAME/backup/2015-10-06T181356Z/2015-10-06T181356Z-full/bucket-simplehms/node-167.114.225.106%3A8091/data-0000.cbb,2015-10-06T18:21:07.929552Z,2015-10-06T18:21:09.923508Z,+cKX5JW5ddtJgzK34S9A7g==,,38612992,38612992,OK,

even though everything reports the file transfer as OK some of the files (i.e. the one in the log above) do not exist in the Google Storage bucket.

I’ve tried gsutil ls and the Google Developers Console to ensure it’s not a display issue, it’s not visible in either of them - it just shows an empty folder …

Issue Analytics

  • State:closed
  • Created 8 years ago
  • Comments:8 (3 by maintainers)

github_iconTop GitHub Comments

2reactions
gadgetcommented, Jun 9, 2020

similar happens for me, steps: -I create a file with my own user (gsutil cp) | OK -I delete the file from the bucket with my own user (gsutil rm) | OK -a service user tries to create another file with the very same name | gsutil reports all good, but the file never appears in the bucket

1reaction
thobrlacommented, Oct 6, 2015

It turns out this was an issue in the Google Cloud Platform UI where %3A on the URL was improperly being translated into a colon in the URL. Until that’s fixed, the workaround in the UI is to manually percent-encode the percent sign in the URL, i.e., replace %3A with %253A

Read more comments on GitHub >

github_iconTop Results From Across the Web

cp - Copy files and objects | Cloud Storage - Google Cloud
Description. The gsutil cp command allows you to copy data between your local file system and the cloud, within the cloud, and between...
Read more >
Gsutil cp is not copying files - Stack Overflow
I am trying to use the command line on my local machine (anaconda prompt) to download a folder from a google cloud bucket....
Read more >
Download and export monthly reports - Play Console Help
Download reports from Google Cloud Storage. Reports are available from Google Cloud Storage. Reports are generated daily and accumulated in monthly CSV files....
Read more >
gslib/commands/cp.py - external/gsutil/src - Git at Google
if gs://my_bucket/subdir does not exist, this same gsutil cp command will. create objects named like gs://my_bucket/subdir/a/b/c.
Read more >
copy_helper.py - EECS: www-inst.eecs.berkeley.edu
For example, when running the command: gsutil cp file gs://bucket/abc if there's no existing gs://bucket/abc bucket subdirectory we should copy file to the ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found