paginator PaginationConfig PageSize is used for pagination control instead of MaxItems

If there is 20 items and we want to list 5 items per page, the paginator PaginationConfig seems using PageSize to control pagination.

Following works, “Marker” is return when there is more data to be returned, and “IsTruncated” also return True.

import sys
import boto3
iam = boto3.client("iam")
marker = None
while True:
    paginator = iam.get_paginator('list_users')
    response_iterator = paginator.paginate(
        PaginationConfig={
            'PageSize': 5,
            'StartingToken': marker})
    for page in response_iterator:
        print("Next Page : {} ".format(page['IsTruncated']))
        u = page['Users']
        for user in u:
            print(user['UserName'])
    try:
        marker = page['Marker']
        print(marker)
    except KeyError:
        sys.exit()

However, if PaginationConfig use Maxitem, e.g.

 response_iterator = paginator.paginate(
        PaginationConfig={
            'MaxItems': 5,
            'StartingToken': marker})

The list just show 5 items and stop, no Markers returns(and isTruncated return as false) as mentioned in the documentation.

[joguSD]

Looking at the example in the original post, I think this isn’t really how we intended the pagination interface to be used. I also think there might be some confusion between the properties of a PaginationConfig and the low-level service API inputs to control pagination.

The PaginationConfig object is designed such that the field names remain the same regardless of how the API exposes them. That is MaxItems and PageSize passed to the PaginationConfig mean the same thing to every API operation. For example, the list_users operation exposes a ‘MaxItems’ input that controls what the PaginationConfig defines as PageSize. The paginator interface is designed to smooth over these inconsistencies.

The intent of MaxItems is to limit the total number of items returned across all pages. PageSize is intended to limit how many items are on a single page. While the low-level API pagination fields are in the response pages (IsTruncated and Marker) we aren’t really expecting users to be relying on those as that kind of defeats the purpose of the auto-pagination. And in a case like this, where the MaxItems setting truncated the results in the middle of the page, relying on the API outputs to determine if there are more items can lead to missing some items, or missing the rest of the items. If you want to be doing this you should use build_full_result as mentioned before which can handle truncation in the middle of an API responses’ page for you.

I think this might come down to how we communicate the expected behavior of the auto-paginators.

[kyleknap]

This seems like a bug to me, but need to do more researching. In short, it seems if MaxItems is specified and if a marker is not returned from the service API response, it will not populate the NextToken, but it should because the paginator stopped in the middle of the only page given back. Furthermore if after you call paginate() you can call build_full_result and that will get you a NextToken. So I wonder if it is just an issue with the __iter__