Parsing error after X responsive pages

domain-scan’s gather feature queries the Censys API to collect .gov subdomains. The query parsed.subject.common_name:".gov" or parsed.extensions.subject_alt_name.dns_names:".gov" works on API pages 1-100, but errors out on any page ≥ 101:

$ ./gather censys --suffix=.gov --start=101 --end=101 --force --delay=5 --debug --parents=current-federal.csv --censys_id=<id> --censys_key=<key>

Starting new HTTPS connection (1): www.censys.io
"GET /api/v1/account HTTP/1.1" 200 245
Censys query:
parsed.subject.common_name:".gov" or parsed.extensions.subject_alt_name.dns_names:".gov"

Fetching up to 100 records, starting at page 101.
Fetching page 101.
"POST /api/v1/search/certificates HTTP/1.1" 400 116
Traceback (most recent call last):

  File "/home/hmft/Desktop/domain-scan-master/gatherers/censys.py", line 100, in gather
    certs = list(certificate_api.search(query, fields=fields, page=current_page, max_records=page_size))

  File "/usr/local/lib/python3.5/dist-packages/censys/base.py", line 151, in search
    payload = self._post(self.search_path, data=data)

  File "/usr/local/lib/python3.5/dist-packages/censys/base.py", line 98, in _post
    return self._make_call(self._session.post, endpoint, args, data)

  File "/usr/local/lib/python3.5/dist-packages/censys/base.py", line 92, in _make_call
    const=const)

censys.base.CensysException: 400 (es_transport_error): Your search was invalid and could not be parsed.

This same query on the web indicates 7,661 responsive pages:

…but any page ≥ 401 produces an error:

cc: @konklone

[zakird]

I just wanted to note that this error message should now describe the problem and paths forward. Example at https://censys.io/certificates?q=*&page=401.

[konklone]

Thank you!