How to filter by datetime downloaded tweets?

I’ve split in consecutive dates periods my searches with Twarc. I am able to join the files easily with cat file1.jsonl file2.jsonl.

Now what I need is to specifically select certain tweets from the downloaded tweets: from one date and time to another. Is there an easy straight forward way to do it?

Why? I am comparing Twarc downloaded dataset with previously donwloaded data sets (with streaming API) and I need to have the same period.

[numeroteca]

@numeroteca since it seems like you have some facility with R is there a reason why you don’t read in the JSON and filter it yourself before sending it off to twarc2 network ?

Certainly not, I could filter it with R. I haven’t tried, but I guess it should work. I’ll try an report back. I was just trying to make as much as possible with twarc before entering the analysis with R.

[igorbrigadir]

I wonder if we can do both - https://github.com/DocNow/twarc/issues/496#issuecomment-867935406 i had a similar idea for splitting the output. Maybe instead of a plugin it should be in core twarc? Since it won’t require any extra dependencies and will be useful to have.