listing the top level contents of a s3 bucket with Prefix and Delimiter

Apologies for what sounds like a very basic question. In this example from the s3 docs is there a way to list the continents? I was hoping this might work, but it doesn’t seem to:

import boto3

s3 = boto3.resource('s3')
bucket = s3.Bucket('edsu-test-bucket')

for o in bucket.objects.filter(Delimiter='/'):
    print(o.key)

However, the equivalent code using boto2 does seem to work the way I expect:

import boto

s3 = boto.connect_s3()
bucket = s3.get_bucket('edsu-test-bucket')

for o in bucket.list(delimiter='/'):
    print(o.name)

[amatthies]

@edsu I ran into this as well. The “directories” to list aren’t really objects (but substrings of object keys), so I do not expect them to show up in an objects collection. As a quick workaround, I list them via client.list_objects.

For non-public buckets (or buckets that you can explicitly access):

import boto3

s3 = boto3.resource('s3')
bucket = s3.Bucket('edsu-test-bucket')
result = bucket.meta.client.list_objects(Bucket=bucket.name,
                                         Delimiter='/')
for o in result.get('CommonPrefixes'):
    print(o.get('Prefix'))

Would print:

Europe/ North America/

This doesn’t support anonymous calls, though. When I run this, I get an AccessDenied error.

For anonymous calls I haven’t found a way to use a s3 resource at all (so far). But I can call list_objects on a low-level client:

import boto3
import botocore

client = boto3.client('s3',  # region_name='us-east-1',
                      config=botocore.client.Config(signature_version=botocore.UNSIGNED))
result = client.list_objects(Bucket='edsu-test-bucket',
                             Prefix='North America/',
                             Delimiter='/'
                             )
for o in result.get('CommonPrefixes'):
    print(o.get('Prefix'))                   

Not very beautiful, but it prints what I wanted

North America/Canada/ North America/USA/

[kyleknap]

@amatthies is on the right track here. The reason that it is not included in the list of objects returned is that the values that you are expecting when you use the delimiter are prefixes (e.g. Europe/, North America) and prefixes do not map into the object resource interface. If you want to know the prefixes of the objects in a bucket you will have to use list_objects. However, I would suggest to use the pagination interface for this because this will allow you to iterate through all objects in the bucket without having to provide pagination tokens:

import boto3
client = boto3.client('s3')
paginator = client.get_paginator('list_objects')
for result in paginator.paginate(Bucket='edsu-test-bucket', Delimiter='/'):
    for prefix in result.get('CommonPrefixes'):
        print(prefix.get('Prefix'))

As to your question as how to use anonymous clients for resources try the following. You will have to hook into the event system to disable signing:

import boto3

from botocore.handlers import disable_signing

resource = boto3.resource('s3')

resource.meta.client.meta.events.register(
    'choose-signer.s3.*', disable_signing)

I realize that this not documented anywhere. Documenting the event system and things you can do with it is on our list of thing that we want to do.

Based on the conversation, I see the following action items:

1. Improve documentation to use client when you are trying to get prefixes.
2. Document the event system and things you can do with the event system
3. Possibly adding a way to disable signing upon instantiation of the resource.

Let me know what you all think or if there is anything else that should be added to this list.