Should twarc filter write limit warnings to output?

Using filter mode like so: twarc filter aleppo --log aleppo_filter.log > aleppo_filter.json, I collected a few million tweets, and noticed I was having a number of issues using the utilities, mailing ids.py and deduplicate.py when I combine that filter output with search output. Lots of uhoh: 'id_str from ids.py, and I got two different counts:

$ wc -l aleppo_filter.json filter_ids.txt
    6136161 aleppo_filter.json
    6133958 filter_ids.txt
   12270119 total

I added a couple lines to ids.py to print out the offending line numbers, and printed out an offending line:

$ sed -n '4502042p' aleppo_filter.json 
{"limit": {"track": 2530, "timestamp_ms": "1482168932301"}}

That’s from going over the 1% of the over all stream, right? If so, should that be in the output from the filter stream using twarc filter aleppo --log aleppo_filter.log > aleppo_filter.json?

[edsu]

Sure, that would be great.

[ruebot]

…I could clean it up and put it utils if you think that would be useful.