question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

listObjectsV2 causes endless loop > 1000 items

See original GitHub issue

Using the new (default) listObjectsV2 to list objects causes an endless loop when there are more than 1000 items returned by the server and recursive is set to true. The iterator will list the first 1000 items over and over again.

I guess there is some error in the continuation token logic in the java client.

To reproduce, create 1001 items in a bucket and use the simplistic code below to list them:

    private static void testEndlessLoop(){

        MinioClient minioClient = null;
        try {
            minioClient = new MinioClient(
                    "http://myminio.lan:9000",
                    accessKey,
                    accessSecret);


            Iterable<Result<Item>> iterable = minioClient.listObjects(
                    "mybucket",
                    "some/prefix/",
                    true,
                    false);  // if set to true it works, using V1
            

            iterable.forEach(o -> {
                try {
                    System.out.println(o.get().objectName());
                } catch (Exception e) {
                    e.printStackTrace();
                }
            });
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

If you have trouble reproducing it on your end let me know.

Versions minio-java 3.0.4 minio RELEASE.2017-05-05T01-14-51Z

Issue Analytics

  • State:closed
  • Created 6 years ago
  • Comments:5 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
harshavardhanacommented, Jun 19, 2017

@harshavardhana any chance of a new release which contains this fix? While the problem is not really an issue for me (as I switched back to the v1 api), all new users have a non working default configuration as V2 has been made the default.

Sure we will make a new release soon @IsNull - thanks for getting back to us.

0reactions
klingtnetcommented, Jun 20, 2017

This is also an issue for the latest minio-go. See #719 in minio-go.

Read more comments on GitHub >

github_iconTop Results From Across the Web

ListObjectsV2 causes endless loop > 1000 items #719
I used a ceph backend and calls to ListObjectsV2 did not stop for some prefixes, with more than 1000 objects, in fact the...
Read more >
Unable to fetch list of all S3 objects using NodeJs
I've following lambda function which isn't fetching list of objects (more than 1000) in S3 and stuck in infinite loop, resulting lambda ...
Read more >
ListObjectsV2 - Amazon Simple Storage Service
Returns some or all (up to 1000) of the objects in a bucket with each request. ... Causes keys that contain the same...
Read more >
Listing keys in an S3 bucket with Python - alexwlchan
This call only returns the first 1000 keys. API responses have a ContinuationToken field, which can be passed to the ListObjects API to...
Read more >
How to list all the objects in Amazon S3 | Edureka Community
S3Client.ListObjects return only 1000 of objects. How to retrieve list of all existing objects using Amazon C# library?
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found