True cursor pagination for Django

Problem

The default pagination approach used for Django querysets executes a .count() on the query. For tables with hundreds of thousands or millions of rows, this is very slow on Postgres. It also assumes that the set is stable, which somewhat defeats the point of using a cursor based pagination system in the first place.

Possible solution

I released today django-cursor-pagination which provides a paginator class which fulfils the relay cursor connection spec. It is available on PyPI.

Below is a pretty simple approach to using this in Graphene, with inspiration taken from graphene-gae - thanks to @ekampf. It would want some checks around full_args, and possibly a means to explicitly set the ordering. It would also need testing and documenting.

Should we use this in graphene? There are some caveats of using this approach. Most importantly, the ordering fields in my current implementation must uniquely determine the items in the queryset. (Discussion of this restriction). Perhaps we should provide both options?

import graphene
from cursor_pagination import CursorPaginator

def connection_from_cursor_paginated(queryset, args, connection_type, edge_type, pageinfo_type, **kwargs):
    paginator = CursorPaginator(queryset, queryset.query.order_by)
    full_args = dict(args, **kwargs)
    page = paginator.page(**full_args)

    edges = []
    for item in page:
        edge = edge_type(node=item, cursor=paginator.cursor(item))
        edges.append(edge)

    return connection_type(
        edges=edges,
        page_info=pageinfo_type(
            start_cursor=paginator.cursor(page[0]),
            end_cursor=paginator.cursor(page[-1]),
            has_previous_page=page.has_previous,
            has_next_page=page.has_next,
        )
    )


class CursorPaginatedConnection(graphene.relay.types.Connection):
    @classmethod
    def from_list(cls, queryset, args, context, info):
        connection = connection_from_cursor_paginated(queryset, args, connection_type=cls, edge_type=cls.edge_type, pageinfo_type=graphene.relay.PageInfo)
        connection.set_connection_data(queryset)
        return connection


class CursorPaginatedConnectionField(graphene.relay.ConnectionField):
    def __init__(self, *args, **kwargs):
        kwargs['connection_type'] = kwargs.pop('connection_type', CursorPaginatedConnection)
        super(CursorPaginatedConnectionField, self).__init__(*args, **kwargs)

[syrusakbary]

I think is worth to take a closer look into this approach.

Also, having an order that strictly determine all the items in the queryset is somewhat essential for pagination.

My ideas here are:

• Use CursorPaginatedConnection when possible (if a query order its defined).
• Use the current behavior otherwise.

Let’s play with it in a branch that could let us see the edge cases

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.