question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Use iterator for s3 object collection

See original GitHub issue
>>> objects
s3.Bucket.objectsCollection(s3.Bucket(name='my-project'), s3.ObjectSummary)

>>> print(next(objects))
TypeError: 's3.Bucket.objectsCollection' object is not an iterator

https://wiki.python.org/moin/Iterator

It does support iter(objects) wrapping, e.g.

>>> obj_iter = iter(objects)
>>> obj_iter
<generator object ResourceCollection.__iter__ at 0x7f57efbff660>

But why is this necessary?

Issue Analytics

  • State:open
  • Created 5 years ago
  • Reactions:3
  • Comments:6 (2 by maintainers)

github_iconTop GitHub Comments

2reactions
dazza-codescommented, Mar 21, 2019

Not sure if the latest release already supports this, but it seems like all() should return an iterable.

objects = s3.Bucket(name='test').objects.all()
next(objects) -> s3.ObjectSummary

The docs, e.g. https://boto3.amazonaws.com/v1/documentation/api/latest/guide/collections.html, explicitly indicate that A collection provides an iterable interface to a group of resources.

It seems like the distinction between iterable and iterator is important [1] and what the intention is for the collections. If the intention is to be an iterable and not an iterator, it’s good to go and close this issue at will.

[1] https://www.geeksforgeeks.org/python-difference-iterable-iterator/

1reaction
Aeternitaascommented, Aug 31, 2019

Alternatively, you may also use something like:

...
bucket_iter = iter(objects)
next(bucket_iter)
...

This works since all Generators (which are used in boto3 to yield API results) are Iterators and all Iterators are Iterables.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Iterating Over Your Objects with Amazon S3 - AWS
The S3Objects and S3Versions classes allow you to easily iterate over objects and object versions in your Amazon S3 buckets, without having to ......
Read more >
How to iterate over files in an S3 bucket? - Stack Overflow
stored in an S3 bucket, and I would like to iterate over them (e.g. in a for loop) to extract data from them...
Read more >
Collections reference — Boto3 Docs 1.26.33 documentation
Get all items from the collection, optionally with a custom page size and item count limit. This method returns an iterable generator which...
Read more >
Using the Boto3 S3 Service Resource - VAST Data
Some resources have collections, which are groups of resources , allowing you to iterate, filter and manipulate resources. For example, a bucket ......
Read more >
List S3 contents and download links in Rails - Mintbit
First, we initialize an Aws::S3::Bucket instance using the bucket name. Next, we call the objects method which returns a collection of all the ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found