question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

list_objects_v2's MaxKeys not working

See original GitHub issue

I’ve 16000 records in s3 bucket’s prefixed location. Used below code to listdown all keys in the prefixed location:

s3Contents =  client.list_objects_v2(
        Bucket=config['aws']['src']['bucket'],
        MaxKeys=50000,
        Prefix=config['aws']['src']['dir'])

However, it’s only listing default behaviour of listing 1000 records although specified 50000 as max keys.

Hence, using NextContinuationToken, I’ve to get the full list. Is this expected behaviour or a bug?

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:7 (2 by maintainers)

github_iconTop GitHub Comments

2reactions
AIGyancommented, Feb 11, 2020

@kyleknap Thanks for your response. Could you please mention how to list more than 1000 object using list_objects ?

0reactions
roryboksercommented, Aug 4, 2021

Use the pagination:

--starting-token (string)
A token to specify where to start paginating. This is the NextToken from a previously truncated response.
For usage examples, see Pagination in the AWS Command Line Interface User Guide .

From: https://docs.aws.amazon.com/cli/latest/reference/s3api/list-objects-v2.html

Recursively make the call with pagination until there is no Truncation.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Does Amazon S3 have a limit to MaxKeys when calling ...
I tried setting the MaxKeys setting to 1,000 but it still pulls 1,080 results. My code: $iterator = $s3->getIterator('ListObjects', ...
Read more >
ListObjectsV2 - Amazon Simple Storage Service
These rolled-up keys are not returned elsewhere in the response. Each rolled-up result counts as only one return against the MaxKeys value. Type:...
Read more >
Amazon S3 ListObjects API
Specifies the key to start with when listing objects in a bucket. max-keys. Sets the maximum number of keys returned in the response....
Read more >
Object Storage Service:List objects - Alibaba Cloud
List objects,Object Storage Service:You can call GetBucket (ListObjects) in ... problem while trying to communicate with OSS, " + "such as not being...
Read more >
list-objects-v2 — AWS CLI 2.9.8 Command Reference
If the number of results exceeds that specified by MaxKeys, all of the results might not be returned. Contents -> (list). Metadata about...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found