CommandException: No URLs matched when passing URLs to rm from stdin

I’m trying to use gsutil rm -I and pass a list of URLs to delete through stdin.

For an existing directory in, say, gs://test-bucket/test-dir, these are some commands I’ve tried:

# verify directory exists
$ gsutil ls -d gs://test-bucket/test-dir
gs://test-bucket/test-dir/

$ echo "gs://test-bucket/test-dir" | gsutil -m rm -r -I
CommandException: No URLs matched

$ echo gs://test-bucket/test-dir | gsutil -m rm -r -I
CommandException: No URLs matched

$ gsutil -m rm -r -I <<< "gs://test-bucket/test-dir"
CommandException: No URLs matched

$ gsutil ls -d gs://test-bucket/test-dir | gsutil -m rm -r -I
CommandException: No URLs matched

Am I missing something here?

[yaseenkhanmohmand]

Seeing the same problem. any suggested solutions?

[dilipped]

Sorry for the delay in response. Our team is currently occupied with other priorities and does not have the bandwidth to address this issue at the moment. However, I did some investigation for future reference.

This seems to be happening because the url_strs gets iterated twice, once here https://github.com/GoogleCloudPlatform/gsutil/blob/d8626ae0ec4b4dc9fd729f115cdeefced4680cb5/gslib/commands/rm.py#L269 if recursion is requested, and next it gets passed to the NameExpansionIterator https://github.com/GoogleCloudPlatform/gsutil/blob/d8626ae0ec4b4dc9fd729f115cdeefced4680cb5/gslib/commands/rm.py#L288

So essentially, we are trying to iterate over the iterator twice and hence on the second instance, we get an empty iterator.

The easy fix would be to convert the iterator to a list, i.e changing https://github.com/GoogleCloudPlatform/gsutil/blob/d8626ae0ec4b4dc9fd729f115cdeefced4680cb5/gslib/commands/rm.py#L252 to

url_strs = [url for url in StdinIterator()]

But this can affect users who have really long list coming from stdin or users who are already using this feature in a pipeline and not really using the -r with -I. Note that this will only affect your if you are using -r and -I together.

The ideal fix would be to remove the recursion special case and instead handle the bucket deletion based on the NameExpansionIterator result itself.

A workaround would be something that is suggested here https://github.com/GoogleCloudPlatform/gsutil/issues/490#issuecomment-364611242

Alternatively, you can avoid using recursion (-r option) and pass in the list

gsutil ls gs://my_bucket/** | gsutil -m rm -I

Note that the above command will empty the bucket, but will not remove the bucket and you will have to run a separate command to remove it.