Max retries exceeded

I run a code twarc search blah-blah > search.jsonl, I already got almost 400 thousands tweets. Then, I got the error below:

ERROR caught connection error HTTPSConnectionPool(host=‘api.twitter.com’, port=443): Max retries exceeded with url: /1.1/search/tweets.json?count=100&q=%23HurricaneDorian2019+OR+%23Dorian2019+OR+%23HurricaneDorian&include_ext_alt_text=true&lang=en&result_type=recent&tweet_mode=extended&max_id=1167786952499900416 (Caused by NewConnectionError(‘<urllib3.connection.VerifiedHTTPSConnection object at 0x103cb62b0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known’)) on 162 try

This error had been reported before in twarc.log, but this time I got the error below in command line: RecursionError: maximum recursion depth exceeded in __instancecheck__

I think this is because of the recursive methods in decorates.py function, in rate_limit class. Is it correct? If so, is there any way to fix it?

[edsu]

Interesting! Maybe it was a temporary network error?

One thing you can do to pick up where you left off (without having to collect all 400k tweets is to get the ID of the last tweet you were able to collect and then pick up there. One handy way of getting the last tweet id in your file would be to use jq.

tail -1 tweets.jsonl | jq .id_str 

Then you can tell twarc to search for tweets earlier than that tweet:

twarc search '#HurricaneDorian2019 OR #Dorian2019 OR #HurricaneDorian' --max_id 1169272246537457671 > tweets2.jsonl

[pbabvey]

All right. Of course. Thanks!