Utility for paginating through a BOTO3 dynamodb result for resource/tables

Is your feature request related to a problem? Please describe.

When fetching a record set that requires pagination you have to implement something quite awkward. And if don’t naively, you could run into memory issues.

dynamodb.Table = boto3.resource("dynamodb")
large_table = dynamodb.Table("TableWithLotsOfData")

# Scan or Query with lots of data
response = large_table.scan()
items = response["Items"]

while "LastEvaluatedKey" in response:
        # Add ExclusiveStartKey to the original operation
	  response = table.scan(ExclusiveStartKey=response["LastEvaluatedKey"])
	  items.update(response["Items"])

or use DynamoDB.Paginator.Query.paginate

client = boto3.client("dynamodb")
paginator = client.get_paginator("scan")
params = {}

for page in paginator.paginate(params):
    # do something

Describe the solution you’d like

Build a custom iterator (much like how the Java AWS SDK has), which can automatically fetch more data as needed.

Describe alternatives you’ve considered

• Building a small local library for this
• doing the naive example.
• Log an issue with boto3 library

Additional context

• Java AWS SDK V1 PaginatedQueryList

[ran-isenberg]

@michaelbrewer I’ve been using PynamoDB for the past year and can confirm it’s awesome and makes life much easier with DynamdoDB.

[heitorlessa]

Thanks for opening that.

That’s been an old request asking boto to port the pagination that boto3 DynamoDB client has onto the resource

https://github.com/boto/boto3/issues/2039

This is one among other suboptimal UX issues that makes Pynamodb be a better solution

https://pynamodb.readthedocs.io/en/latest/