`twarc2 counts --archive` returning incomplete results

I’m running twarc2 counts with the Academic Track of the Twitter API, and for some reason it’s returning incomplete results (only going back a few years) with the following query:

!twarc2 counts "(\"this is how\" \"not with a\" but) OR (\"not with a bang\") OR (\"not with a\" \"but a whimper\") OR (\"not with a\" \"but with a whimper\") OR (\"this is the way\" \"not with a\" but)" --archive --granularity day --csv > tweets.csv

(For reference I’m doing a project about T.S. Eliot’s “not with a bang but a whimper.”)

When I check twarc.log, I can see that an empty page is returned, which is interpreted as the end of the search:

2022-02-28 15:59:21,700 INFO Retrieved an empty page of results
2022-02-28 15:59:21,700 INFO No more results for search.

[SamHames]

I did put a workaround together in the linked #604 - it turned out to be easier than I thought it was going to be, as long as start-time is specified in the query we can check if we’ve reached that point or not.

@melaniewalsh - you’re welcome to test against that branch, or if it would be easier for you I can push a pre-release to PyPI on Monday.

[edsu]

I greatly appreciate the API detective work here @melaniewalsh & @SamHames – we all benefit from your efforts here.

I tried running @melaniewalsh’s query above and it works now 🥳

(twarc) ➜  twarc git:(workaround_counts_API_zero_count) ✗ twarc2 counts "(\"this is how\" \"not with a\" but) OR (\"not with a bang\") OR (\"not with a\" \"but a whimper\") OR (\"not with a\" \"but with a whimper\") OR (\"this is the way\" \"not with a\" but)" --archive --granularity day --csv tweets.csv
100%|█████████████████| Processed 15 years/15 years [16:33<00:00, 354018 tweets total ]