Provide a more convenient way to paginate via the Python package

Currently, the way to paginate searches is to get the serpapi_pagination.current and increase the offset or start parameters in the loop. Like with regular HTTP requests to serpapi.com/search without an API wrapper.

import os
from serpapi import GoogleSearch

params = {
    "engine": "google",
    "q": "coffee",
    "tbm": "nws",
    "api_key": os.getenv("API_KEY"),
}

search = GoogleSearch(params)
results = search.get_dict()

print(f"Current page: {results['serpapi_pagination']['current']}")

for news_result in results["news_results"]:
    print(f"Title: {news_result['title']}\nLink: {news_result['link']}\n")

while 'next' in results['serpapi_pagination']:
    search.params_dict[
        "start"] = results['serpapi_pagination']['current'] * 10
    results = search.get_dict()

    print(f"Current page: {results['serpapi_pagination']['current']}")

    for news_result in results["news_results"]:
        print(
            f"Title: {news_result['title']}\nLink: {news_result['link']}\n"
        )

A more convenient way for an official API wrapper would be to provide some function like search.paginate(callback: Callable) which will properly calculate offset for the specific search engine and loop through pages until the end.

import os
from serpapi import GoogleSearch

def print_results(results):
  print(f"Current page: {results['serpapi_pagination']['current']}")

  for news_result in results["news_results"]:
    print(f"Title: {news_result['title']}\nLink: {news_result['link']}\n")

params = {
    "engine": "google",
    "q": "coffee",
    "tbm": "nws",
    "api_key": os.getenv("API_KEY"),
}

search = GoogleSearch(params)
search.paginate(print_results)

@jvmvik @hartator What do you think?

[jvmvik]

Actually we do not need a callback because SerpApi backend does already offer proper pagination.

# initialize the search
search = GoogleSearch({"q": "Coffee", "location": "Austin,Texas"})
# to get 2 pages
start = 0
end = 20
# create a python generator
pages = search.pagination(start, end)
print("display generated")
urls = []
# fetch one result per iteration of the for loop
for result in pages:
	urls.append(result['serpapi_pagination']['next'])

self.assertEqual(len(urls), 2)
self.assertTrue("start=10" in urls[0])
self.assertTrue("start=20" in urls[1])

see commit: f9a470a9efd8c8956cb84597b4c33136670e3abb

[jvmvik]

thanks for your feedbacks!

@ilyazub code improved to handle missing “serpapi_pagination” field on the last page. @kikohs README updated with information on pagination support.

Library released on pypi