listObjectsV2 causes endless loop > 1000 items
See original GitHub issueUsing the new (default) listObjectsV2 to list objects causes an endless loop when there are more than 1000 items returned by the server and recursive is set to true
. The iterator will list the first 1000 items over and over again.
I guess there is some error in the continuation token logic in the java client.
To reproduce, create 1001 items in a bucket and use the simplistic code below to list them:
private static void testEndlessLoop(){
MinioClient minioClient = null;
try {
minioClient = new MinioClient(
"http://myminio.lan:9000",
accessKey,
accessSecret);
Iterable<Result<Item>> iterable = minioClient.listObjects(
"mybucket",
"some/prefix/",
true,
false); // if set to true it works, using V1
iterable.forEach(o -> {
try {
System.out.println(o.get().objectName());
} catch (Exception e) {
e.printStackTrace();
}
});
} catch (Exception e) {
e.printStackTrace();
}
}
If you have trouble reproducing it on your end let me know.
Versions
minio-java 3.0.4
minio RELEASE.2017-05-05T01-14-51Z
Issue Analytics
- State:
- Created 6 years ago
- Comments:5 (4 by maintainers)
Top Results From Across the Web
ListObjectsV2 causes endless loop > 1000 items #719
I used a ceph backend and calls to ListObjectsV2 did not stop for some prefixes, with more than 1000 objects, in fact the...
Read more >Unable to fetch list of all S3 objects using NodeJs
I've following lambda function which isn't fetching list of objects (more than 1000) in S3 and stuck in infinite loop, resulting lambda ...
Read more >ListObjectsV2 - Amazon Simple Storage Service
Returns some or all (up to 1000) of the objects in a bucket with each request. ... Causes keys that contain the same...
Read more >Listing keys in an S3 bucket with Python - alexwlchan
This call only returns the first 1000 keys. API responses have a ContinuationToken field, which can be passed to the ListObjects API to...
Read more >How to list all the objects in Amazon S3 | Edureka Community
S3Client.ListObjects return only 1000 of objects. How to retrieve list of all existing objects using Amazon C# library?
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Sure we will make a new release soon @IsNull - thanks for getting back to us.
This is also an issue for the latest
minio-go
. See #719 inminio-go
.