Replicate directory structure in bucket using gsutil cp with wildcards

This is essentially a feature request for the question I posted on stackoverflow.

Let’s say I have the following folders in my bucket:

2011/03/11/a/b/c
2012/04/11/c
2013/04/11/f

When I copy directories from my bucket using:

$ gsutil cp -r gs://my-bycket .

it creates the folders (2011/03/11/a/b/c etc) on my local harddrive and then place the files under these folders (which is what I want). The problem is that I don’t want to copy ALL files in these folders so I use a wildcard like this instead:

gsutil cp -r "gs://my-bucket/**/name-????.gz" .

gsutil starts copying all the files correctly but it flattens out the folder structure and place each file directly in my current directory which is not what I want.

I’d like a flag that allows gsutil to create the directory structure for each file when doing this. Would this be feasible?

[leo-p]

Last time I checked neither features had been implemented (correct me if I’m wrong), but you don’t actually need it.

Using the -x flag from the gsutil rsync command, you can actually exclude all files that don’t match (instead of just the one that match). All you need to do is use a negative lookahead (?!...).

For instance, if you would like to match all JSON files, you could use a regular expression such as:

.*\.json$

Now you just need to add the negative lookahead:

^(?!.*\.json$).*

Combined with the -x flag, this will sync all JSON files in your gs bucket to a local directory:

gsutil rsync -r -x '^(?!.*\.json$).*' gs://mybucket mydir

Example

$ gs://mybucket
├── dir1
│   ├── 1.json
│   ├── 2.json
│   └── img.jpg
└── dir2
    ├── 3.json
    └── random.txt
$ mkdir mydir
$ gsutil rsync -r -x '^(?!.*\.json$).*' gs://mybucket mydir
$ mydir
├── dir1
│   ├── 1.json
│   └── 2.json
└── dir2
    └── 3.json

[jsonbrooks]

Any update on this?